Will's Software Journal

Rust benchmarking with Criterion

Wed, 03 Jul 2024 00:00:00 +0000

I was recently looking into benchmarking some Rust code and thought I would write up how to do so with the criterion library. Code for this post is available here at GitHub.

For this toy example we'll be writing code to determine the distance between 2 points. This is relatively easy to do with the pythagorean theorem, but the downside is you will need to use a relatively slow square root calculation to do so.

Many times you don't need the exact distance. Say you have a point and a vector of points and want to know which one in the vector is the closest - in this case what you care about is the relative distance each point is, and can skip the square root and use distance squared.

So how much faster would it be to use squared distance? Lets find out.

Create a library

Lets create a library and add criterion as a dependency.

$ cargo new --lib rust_benchmarking_example
$ cd rust_benchmarking_example

Now lets add criterion as a dev-dependency

cargo add criterion --dev --features html_reports

Your Cargo.toml file should now look something like this

[package]
name = "rust_benchmarking_example"
version = "0.1.0"
edition = "2021"

[dependencies]

[dev-dependencies]
criterion = { version = "0.5.1", features = ["html_reports"] }

Code

Open src/lib.rs and add the following code

pub struct Vector2 {
    pub x: f64,
    pub y: f64,
}

impl Vector2 {
    pub fn new(x: f64, y: f64) -> Vector2 {
        Vector2 { x, y }
    }

    pub fn distance(&self, rhs: &Vector2) -> f64 {
        let x = self.x - rhs.x;
        let y = self.y - rhs.y;
        (x * x + y * y).sqrt()
    }

    pub fn distance_squared(&self, rhs: &Vector2) -> f64 {
        let x = self.x - rhs.x;
        let y = self.y - rhs.y;
        x * x + y * y
    }
}

Unit tests

Lets make sure everything is working as expected, add the following tests to src/lib.rs.

#[cfg(test)]
mod tests {
    use super::*;

    fn approximately(lhs: f64, rhs: f64) -> bool {
        (lhs - rhs).abs() < f64::EPSILON
    }

    #[test]
    fn distance_test() {
        let a = Vector2::new(1.0, 5.0);
        let b = Vector2::new(-2.0, 1.0);

        let result = a.distance(&b);
        assert!(approximately(result, 5.0), "{result} was not equal to 5.0");

        let result = b.distance(&a);
        assert!(approximately(result, 5.0), "{result} was not equal to 5.0");
    }

    #[test]
    fn distance_squared_test() {
        let a = Vector2::new(0.0, 0.0);
        let b = Vector2::new(3.0, 4.0);

        let result = a.distance_squared(&b);
        assert!(
            approximately(result, 25.0),
            "{result} was not equal to 25.0"
        );

        let result = b.distance_squared(&a);
        assert!(
            approximately(result, 25.0),
            "{result} was not equal to 25.0"
        );
    }
}

Create a benchmark

Now lets create a benchmark with criterion, these files are put outside of the src directory in a directory called benches that you will need to create. Lets do that and also create a file in it called distance.rs

At this point your directory structure should look like this

$ tree
.
├── Cargo.lock
├── Cargo.toml
├── benches
│   └── distance.rs
├── src
│   └── lib.rs

Add the following to your Cargo.toml file.

[[bench]]
name = "distance"
harness = false

and the following code to benches/distance.rs

use criterion::{criterion_group, criterion_main, Criterion};
use rust_benchmarking_example::*;

pub fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("distance", |bench| {
        bench.iter(|| {
            let a = Vector2::new(0.0, 0.0);
            let b = Vector2::new(3.0, 4.0);
            a.distance(&b);
        })
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

You can then run the benchmark from the command like with cargo bench which will run the code a number of times

In the output look for a line like:

distance                time:   [934.09 ps 935.48 ps 937.28 ps]

Which tells you

934.09 ps - fastest benchmark
935.48 ps - average benchmark
937.28 ps - slowest benchmark

The results are cached, so if you run cargo bench again it will compare the results against the most recently run ones, here we can see that there was no real change detected - which makes sense as other than some background noise on your computer we are running it on the same code.

distance                time:   [934.73 ps 935.46 ps 936.38 ps]
                        change: [-0.1119% +0.0493% +0.2134%] (p = 0.55 > 0.05)
                        No change in performance detected.

And since we enabled the html_reports feature we can see them by opening target/criterion/report/index.html in a web browser, where you should see something that looks like this

Benchmark squared distance implementation

Now lets benchmark our distance squared code, change

c.bench_function("distance", |bench| {
    bench.iter(|| {
        let a = Vector2::new(0.0, 0.0);
        let b = Vector2::new(3.0, 4.0);
        a.distance(&b);
    })
});

c.bench_function("distance", |bench| {
    bench.iter(|| {
        let a = Vector2::new(0.0, 0.0);
        let b = Vector2::new(3.0, 4.0);
        a.distance_squared(&b);
    })
});

and run cargo bench again. On my machine I get this error

Benchmarking distance: AnalyzingCriterion.rs ERROR: At least one measurement of benchmark distance took zero time per iteration. This should not be possible. If using iter_custom, please verify that your routine is correctly measured.

It appears that the distance_squared() implementation is so fast it effectively takes zero time to run and can't be measured. Which I suppose makes sense as the much slower version using a square root took picoseconds to run.

If you really want to have this complete you could artificially make it slower by disabling compiler optimizations when benchmarking by adding this to your Cargo.toml

[profile.bench]
opt-level = 0

but you generally wouldn't want to do so since taking measurements on a dev build won't be very useful. If you were to do so, you would see output like

distance                time:   [9.9843 ns 10.000 ns 10.017 ns]

distance                time:   [9.3626 ns 9.3730 ns 9.3848 ns]
                        change: [-6.2155% -6.0317% -5.8404%] (p = 0.00 < 0.05)
                        Performance has improved.

Alternatively you could also do a benchmark run where only the the a.distance(&b); call happens and another where it calls that and a.distance_squared(&b); and you can see that the distance_squared call does not add any noticeable overhead.

But realistically the code in this toy example is probably so fast that it is not a great candidate for benchmarking.

Clojure Performace and the Billion Row Challenge

Sun, 21 Jan 2024 00:00:00 +0000

I recently discovered The One Billion Row Challenge and thought i'd give it a shot using Clojure. TLDR: the solution I ended up with is ~41% faster than the Java baseline provided by the challenge.

The problem is described as:

Write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There's just one caveat: the file has 1,000,000,000 rows!

The text file has a simple structure with one measurement value per row:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6

The challenge has a GitHub repo with submissions and leaderboards.

For most of this post i'll be using measurements against 100 million rows because 1 billion takes too long to test solutions, but I will have results on the full data set at the end of this post.

When I run the provided Java baseline on my machine (a Mac mini with a 6 core Intel Core i5 and 16GB of RAM) to establish a baseline I get:

$ time ./calculate_average_baseline.sh
20.80s user 0.65s system 102% cpu 20.948 total

So we are trying to beat roughly 21 seconds.

To simplify the examples, I'm not actually calculating the average, but the data is there to do so, and it would be relatively straightforward/fast as there are only 413 weather stations in the resulting data set.

Attempt 1: 1 minute 57 seconds

This is a relatively straightforward Clojure implementation. But with a 1:57 runtime, not at all competitive with the 21 second baseline.

(ns attempt1)

(defn do-calc [acc row]
  (let [[station measurement] (clojure.string/split row #";")
        measurement (Double/parseDouble measurement)
        station-data (get acc station {:min measurement :max measurement :sum 0.0 :count 0})
        new-data (-> station-data
                     (update :min #(min % measurement))
                     (update :max #(max % measurement))
                     (update :sum #(+ % measurement)) ;; This potentially could overflow, but ignoring for the purposes of this blog post
                     (update :count inc))]
    (assoc acc station new-data)))

(defn run [_opts]
  (with-open [rdr (clojure.java.io/reader "./measurements.txt")]
    (->> (line-seq rdr)
         (reduce do-calc {}))))

Lets profile the code with clj-async-profiler.

It looks like a lot of time was spent in Clojure functions like clojure.core/update, clojure.string/split, and clojure.core/assoc etc

Attempt 2 - 1 minute 37 seconds

Lets try using a transient map for the accumulator in the reduce function

(ns attempt2)

(defn do-calc [acc row]
  (let [[station measurement] (clojure.string/split row #";")
        measurement (Double/parseDouble measurement)
        station-data (get acc station {:min measurement :max measurement :sum 0.0 :count 0})
        new-data (-> station-data
                     (update :min #(min % measurement))
                     (update :max #(max % measurement))
                     (update :sum #(+ % measurement))
                     (update :count inc))]
    (assoc! acc station new-data)))

(defn run [_opts]
   (with-open [rdr (clojure.java.io/reader "./measurements.txt")]
     (->> (line-seq rdr)
          (reduce do-calc (transient {}))
          (persistent!))))

Better, but at 1:37, not much better! Even though we're using a transient map as an accumulator in the reduce, we're still using persistent maps as values in the transient map.

We could try using a transient map with transient maps as keys, but given how transients work in practice that approach seems fraught with peril.

Attempt 3 - Did not finish

Lets try using a Java array to store results instead of a Clojure persistent hash map.

(ns attempt3)

(defn do-calc [acc row]
  (let [[station measurement] (clojure.string/split row #";")
        measurement (Double/parseDouble measurement)
        station-data (get acc station (double-array [measurement measurement 0.0 0.0]))
        min-measurement (aget station-data 0)
        max-measurement (aget station-data 1)
        sum-measurement (aget station-data 2)
        count-measurement (aget station-data 3)]
    (aset station-data 0 (min min-measurement measurement))
    (aset station-data 1 (max max-measurement measurement))
    (aset station-data 2 (+ sum-measurement measurement))
    (aset station-data 3 (+ count-measurement 1.0))

    (assoc! acc station station-data)))

(defn run [_opts]
  (time
   (with-open [rdr (clojure.java.io/reader "/Users/wtf/src/open-source/1brc/measurements.txt")]
     (->> (line-seq rdr)
          (reduce do-calc (transient {}))
          (persistent!)))))

This one was incredibly slow, I had to force quit it before it finished. What is going on here?

If we add the expression

(set! *warn-on-reflection* true)

and rerun, we see a number of lines on stdout that look like:

Reflection warning, attempt3.clj:9:25 - call to static method aget on clojure.lang.RT can't be resolved (argument types: java.lang.Object, int).

Reflection in the Java interop is incredibly slow! Let fix that.

Attempt 4 - 1 minute 6 seconds

We can add type hints to the previous solution like so:

(ns attempt4)

(set! *warn-on-reflection* true)

(defn do-calc [acc ^String row]
  (let [[station measurement] (clojure.string/split row #";")
        measurement (Double/parseDouble measurement)
        station-data ^doubles (get acc station (double-array [measurement measurement 0.0 0.0]))
        min-measurement (aget station-data 0)
        max-measurement (aget station-data 1)
        sum-measurement (aget station-data 2)
        count-measurement (aget station-data 3)]

    (aset station-data 0 ^double (min min-measurement measurement))
    (aset station-data 1 ^double (max max-measurement measurement))
    (aset station-data 2 ^double (+ sum-measurement measurement))
    (aset station-data 3 ^double (+ count-measurement 1.0))
    (assoc! acc station station-data)))

(defn run [_opts]
  (with-open [rdr (clojure.java.io/reader "/Users/wtf/src/open-source/1brc/measurements.txt")]
    (->> (line-seq rdr)
         (reduce do-calc (transient {}))
         (persistent!))))

This one is about 30 seconds better than attempt 2, but still significantly slower than the 21 second Java baseline.

Attempt 5 - 45 Seconds

Lets try some optimizations: manually loop instead of reduce, use Java's String split instead of clojure.string/split and don't destructure the results.

(ns attempt5)

(set! *warn-on-reflection* true)

(defn do-calc [acc ^String row]
  (let [split-row (.split row ";")
        station ^String (aget split-row 0)
        measurement ^String (aget split-row 1)
        measurement (Double/parseDouble measurement)
        station-data ^doubles (get acc station (double-array [measurement measurement 0.0 0.0]))
        min-measurement ^double (aget station-data 0)
        max-measurement ^double (aget station-data 1)
        sum-measurement ^double (aget station-data 2)
        count-measurement ^double (aget station-data 3)]
    (aset station-data 0 ^double (min min-measurement measurement))
    (aset station-data 1 ^double (max max-measurement measurement))
    (aset station-data 2 ^double (+ sum-measurement measurement))
    (aset station-data 3 ^double (+ count-measurement 1.0))
    (assoc! acc station station-data)))

(defn run [_opts]
  (let [reader (java.io.BufferedReader. (java.io.FileReader. "./measurements.txt"))]
    (loop [line (.readLine reader)
           acc (transient {})]
      (if line
        (recur (.readLine reader) (do-calc acc line))
        (persistent! acc)))))

Around 20 seconds faster, but we're still twice the runtime of the Java baseline!

Looking at a flamegraph we seem to be spending a lot of time in calls to clojure.lang.RT.get and clojure.core/assoc!

Attempt 6 - 30 seconds

Lets try replacing the transient Clojure hash map in the accumulator with a java.util.HashMap

(ns attempt6)

(set! *warn-on-reflection* true)

(defn do-calc [^java.util.HashMap acc ^String row]
  (let [split-row (.split row ";")
        station ^String (aget split-row 0)
        measurement ^String (aget split-row 1)
        measurement (Double/parseDouble measurement)
        station-data ^doubles (.getOrDefault acc station (double-array [measurement measurement 0.0 0.0]))
        min-measurement ^double (aget station-data 0)
        max-measurement ^double (aget station-data 1)
        sum-measurement ^double (aget station-data 2)
        count-measurement ^double (aget station-data 3)]
    (aset station-data 0 ^double (min min-measurement measurement))
    (aset station-data 1 ^double (max max-measurement measurement))
    (aset station-data 2 ^double (+ sum-measurement measurement))
    (aset station-data 3 ^double (+ count-measurement 1.0))
    (.put acc station station-data)
    acc))

(defn run [_opts]
  (let [reader (java.io.BufferedReader. (java.io.FileReader. "./measurements.txt"))]
    (loop [line (.readLine reader)
           acc (java.util.HashMap.)]
      (if line
        (recur (.readLine reader) (do-calc acc line))
        acc))))

At 30 seconds we're getting toward respectable compared with the baseline. However we've gotten so deep into Java interop and away from idiomatic Clojure code that it would probably make more sense to just write this solution in pure Java.

Also, maybe we should next try a solution that isn't effectively single threaded.

Attempt 7: core.async

Taking the code from attempt 6, but using core.async to fanout to multiple channels ultimately did not go anywhere, we're largely CPU bound and core.async seemed to just add overhead.

I'm tempted to try this again in the future to see if I can make it faster. Not sharing the code as I never got it to a place I was happy with and didn't want to spend time optimizing a solution that was unlikely to be the best.

Attempt 8: Reducers - 27 seconds

This time we'll go back to using code that looks more like typical Clojure and leverage reducers. The fold function reduces a collection using a (potentially parallel) reduce-combine strategy. Meaning I can easily use all 6 cores on my machine!

One problem with fold is that it should be used when source data can be generated and held in memory, which is not necessarily the case with billions of records. However, we can use the iota library to create a seq of the file which is tuned for reducers and can handle files larger than available memory.

(ns attempt8
  (:require [clojure.core.reducers :as r]
            [iota :as iota]))

(set! *warn-on-reflection* true)

(defn merge-counts
  ([] {})
  ([x y] (merge-with (fn [[x-min x-max x-sum x-count]
                          [y-min y-max y-sum y-count]]
                       [(min x-min y-min) (max x-max y-max) (+ x-sum y-sum) (+ x-count y-count)]) x y)))

(defn do-calc
  ([acc ^String row]
   (let [[station measurement] (clojure.string/split row #";")
         measurement (Double/parseDouble measurement)
         [cur-minimum cur-maximum cur-sum cur-count] (get acc station [measurement measurement 0.0 0])
         new-data [(min measurement cur-minimum) (max measurement cur-maximum) (+ measurement cur-sum) (inc cur-count)]]
     (assoc acc station new-data))))

(defn run [_opts]
  (->> (iota/seq "./measurements.txt")
       (r/fold merge-counts do-calc)
       println))

27 seconds isn't bad for a first attempt with this approach.

Attempt 9: Reducers with optimizations - 9.6 seconds

Lets try the previous attempt but with optimizations used in previous attempts.

(ns attempt9
  (:require [clojure.core.reducers :as r]
            [iota :as iota]))

(set! *warn-on-reflection* true)

(defn merge-counts
  ([] (java.util.HashMap.))
  ([^java.util.HashMap x
    ^java.util.HashMap y]
   ;; Convert to clojure.lang.PersistentHashMap so we can use merge-with.
   ;; Probably could be optimized to directly merge the Java HashMaps
   ;; but reducer merges should be relatively infrequent, so probably not worth optimizing
   (let [x (into {} x)
         y (into {} y)
         ^clojure.lang.PersistentHashMap result (merge-with (fn [[x-min x-max x-sum x-count]
                                                                 [y-min y-max y-sum y-count]]
                                                              [(min x-min y-min) (max x-max y-max) (+ x-sum y-sum) (+ x-count y-count)]) x y)]
     (java.util.HashMap. result))))

(defn do-calc
  ([^java.util.HashMap acc ^String row]
   (let [split-row (.split row ";")
         station ^String (aget split-row 0)
         measurement ^String (aget split-row 1)
         measurement (Double/parseDouble measurement)
         [cur-minimum cur-maximum cur-sum cur-count] (.getOrDefault acc station [measurement measurement 0.0 0])
         new-data [(min measurement cur-minimum) (max measurement cur-maximum) (+ measurement cur-sum) (inc cur-count)]]
     (.put acc station new-data))
   acc))

(defn run [_opts]
  (->> (iota/seq "/Users/wtf/src/open-source/1brc/measurements.txt")
       (r/fold 1028 merge-counts do-calc)
       println))

9.6 seconds isn't super fast, but we are now handily beating the 21 second Java baseline.

Results with 1 Billion rows

Instead of using 100 million rows, we'll finally run against the full 1 billion row dataset:

Entry	Results
Java Baseline	3:47
Mine	2:13
Best competition entry	0:16

We're about a minute faster than the Java baseline.

But we're nowhere close to the 16 second runtime of the best Java entry (or at least best as of the time of writing). That entry describes itself as:

Simple solution that memory maps the input file, then splits it into one segment per available core and uses sun.misc.Unsafe to directly access the mapped memory. Uses a long at a time when checking for collision.

It appears to be using sun.misc.Unsafe to mostly avoid overhead of using java.lang.String (which i'm not sure if it would break if weather stations could contain UTF-8 characters larger than one byte), a custom hash map implementation, a custom implementation of Java's Integer/parseInt, etc. Definitely an impressive runtime, but I almost wonder if it would make more sense to just use C++ with that approach.

Though there are some other interesting entries that are fast and don't use sun.misc.Unsafe like this one that uses the jdk.incubator.vector package to take advantage of Single Instruction Multiple Data (SIMD) parallelism.

Next steps

There are a number of optimizations I could look at to try to get a better time. Looking at a flame graph I think there is a good amount of overhead in the iota code that could be bypassed with a custom solution memory mapping the file into a number of chunks equivalent to the number of processors, but ultimately i'm pretty happy with what I came up with over the course of a Saturday afternoon and probably will call it here.

Pong in ClojureScript

Sat, 23 Apr 2022 00:00:00 +0000

I'm working predominantly with Clojure at work these days, but I wanted to spend some time with ClojureScript, so I figured making a pong clone would be a fun weekend project.

Every time the ball hits a paddle it increases in speed and it's direction is set to a randomized unit vector.
Player two moves up or down depending on if the ball it above or below it
Player one behaves the same when the ball is moving towards it, otherwise it moves to the middle of the screen.

The code is available on GitHub

Create a Clojure AWS Lambda function in Docker and deployed via AWS SAM

Sun, 17 Oct 2021 00:00:00 +0000

In this post I will cover using Clojure to write an AWS Lambda function that will run on a schedule of once per minute. It will be deployed via the AMS SAM CLI tool and use the Docker runtime. The complete code is available at this GitHub repo.

I'll start with a simple Lambda function that just writes a message to stdout once per minute.

The longer term goal is to create a function that will call the National Weather Service API, and then send me a notification if there are any alerts where I live. But that will be a project for another day.

Note that even though Lambda supports Java as a runtime, we'll be running this function in Docker.

While everything works great in the cloud, I ran into some compilation issues running the jar file locally with the sam invoke local command. I think the SAM CLI tool may be doing something unexpected with the java classpath, and could only get it to run without errors using the specific combination of Clojure 1.9 and Java 8. However by using Docker we can easily avoid issues like this.

Project Code

I am using Leiningin for this project, so create a project.clj file with these contents:

(defproject clojure-aws-lambda-example "0.1.0-SNAPSHOT"
  :dependencies [[com.amazonaws/aws-lambda-java-runtime-interface-client "2.0.0"]
                 [org.clojure/clojure "1.10.3"]
                 [org.clojure/data.json "2.4.0"]]
  :repl-options {:init-ns clojure-aws-lambda-example.core}
  :profiles {:uberjar {:aot :all}})

Now lets write our lambda function Create a file called src/clojure_aws_lambda_example/core.clj with these contents:

(ns clojure-aws-lambda-example.core
  (:require [clojure.data.json :as json]
            [clojure.java.io :as io]
            [clojure.spec.alpha :as spec])
  (:gen-class
   :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler]))

;; -------------------- Specs --------------------
;; The json for an inbound scheduled event should look like this
;; {
;;     "version": "0",
;;     "id": "53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa",
;;     "detail-type": "Scheduled Event",
;;     "source": "aws.events",
;;     "account": "123456789012",
;;     "time": "2015-10-08T16:53:06Z",
;;     "region": "us-east-1",
;;     "resources": [
;;         "arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule"
;;     ],
;;     "detail": {}
;; }
;;
;; See https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html
(spec/def ::version string?)
(spec/def ::id string?)
(spec/def ::detail-type string?)
(spec/def ::source string?)
(spec/def ::account string?)
(spec/def ::time string?)
(spec/def ::region string?)
(spec/def ::resources (spec/coll-of string?))
(spec/def ::detail map?)

(spec/def ::scheduled-event
  (spec/keys
   :req-un [::version ::id ::detail-type ::source ::account ::time ::region ::resources ::detail]))

;; -------------------- Request handling --------------------
(defn- stream->scheduled-event
  "Transforms an input stream into a scheduled event "
  [in]
  (json/read (io/reader in) :key-fn keyword))

(defn -handleRequest
  "Implementation for RequestStreamHandler that handles a Lambda Function request"
  [_ input-stream _output-stream _context]
  (println "-handleRequest called with event:" (stream->scheduled-event input-stream)))

Testing

Lets create a simple unit test. Create a file named test/eventbridge-scheduled-event.json with these contents.

{
    "version": "0",
    "id": "53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa",
    "detail-type": "Scheduled Event",
    "source": "aws.events",
    "account": "123456789012",
    "time": "2015-10-08T16:53:06Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule"
    ],
    "detail": {}
}

And a file named test/clojure_aws_lambda_example/core_test.clj with these contents

(ns clojure-aws-lambda-example.core-test
  (:require [clojure.test :refer [deftest is]])
  (:require [clojure-aws-lambda-example.core :as lambda-example]
            [clojure.java.io :as io]
            [clojure.spec.alpha :as spec]))

(deftest test-stream->scheduled-event
  (let [stream (io/input-stream "./test/eventbridge-scheduled-event.json")
        event (lambda-example/stream->scheduled-event stream)]
    (is (spec/valid? ::lambda-example/scheduled-event event))))

In the test we:

Load a sample event from disk.
Pass it to the stream->scheduled-event function.
Use clojure.spec to validate that the return value looks correct.

Note that rather than relying on a single json event, there is a much more we could test using spec's generative testing features, but I will leave that as an exercise for the reader.

Dockerfile

We will also need a Dockerfile. We will use a clojure:openjdk-11-slim-buster image to build the Java jar, and then run the function inside a eclipse-temurin:11-focal container. By using a separate image to build our app, we can use a final image that doesn't have to ship Clojure build tools like lein with it.

Create a file named Dockerfile

# Docker container used to build the Clojure app
FROM clojure:openjdk-11-slim-buster as builder

WORKDIR /usr/src/app

COPY project.clj /usr/src/app/project.clj

# Cache deps so they aren't fetched every time a .clj file changes
RUN lein deps

COPY src/ /usr/src/app/src

RUN lein uberjar

# Build the docker container we will use in the lambda
FROM eclipse-temurin:11-focal

RUN mkdir /opt/app

COPY --from=builder /usr/src/app/target/clojure-aws-lambda-example-0.1.0-SNAPSHOT-standalone.jar /opt/app/app.jar

ENTRYPOINT [ "java", "-cp", "/opt/app/app.jar", "com.amazonaws.services.lambda.runtime.api.client.AWSLambda" ]

CMD ["clojure_aws_lambda_example.core::handleRequest"]

SAM Configuration

Finally we will need a SAM template that will have all the information the SAM CLI tool needs to generate a CloudFormation template. We want to create a Lambda function and a CloudWatch event that fires every minute to trigger a run of the function.

Create a file template.yaml that looks like:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  clojure-aws-lambda-example

  Sample SAM Template for clojure-aws-lambda-example

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 20

Resources:
  ClojureAwsLambdaExampleFunction:
    # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      MemorySize: 256
      Architectures:
        - x86_64
      Events:
        # We want the lambda to run every minute
        # See https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-schedule.html
        MyScheduledEvent:
          Type: Schedule
          Properties:
            Schedule: rate(1 minute)
            Name: MyScheduledEvent
            Description: Event that triggers the lambda every minute
            Enabled: true
    Metadata:
      DockerTag: clojure-aws-lambda-example-v1
      DockerContext: ./
      Dockerfile: Dockerfile

Outputs:
  ClojureAwsLambdaExampleFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt ClojureAwsLambdaExampleFunction.Arn
  ClojureAwsLambdaExampleFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt ClojureAwsLambdaExampleFunctionRole.Arn

Run the app locally

Prior to deploying to the cloud we'll want to run and test on our local machine.

Ensure you have installed the AWS SAM CLI installed, and then build the app.

$ sam build

Run it with

$ sam local invoke --event test/eventbridge-scheduled-event.json

You should see output similar to

Invoking Container created from clojureawslambdaexamplefunction:clojure-aws-lambda-example-v1
Building image.................
Skip pulling image and use local one: clojureawslambdaexamplefunction:rapid-1.33.0-x86_64.

START RequestId: 7dfdf5e7-7580-466b-b3da-c24935837cf0 Version: $LATEST
-handleRequest called with event:  {:detail-type Scheduled Event, :time 2015-10-08T16:53:06Z, :source aws.events, :account 123456789012, :region us-east-1, :id 53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa, :version 0, :resources [arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule], :detail {}}
END RequestId: 7dfdf5e7-7580-466b-b3da-c24935837cf0
REPORT RequestId: 7dfdf5e7-7580-466b-b3da-c24935837cf0  Init Duration: 0.26 ms  Duration: 795.48 ms     Billed Duration: 796 ms Memory Size: 256 MB  Max Memory Used: 256 MB

Deploy to the Cloud

The first time you deploy run this command to generate a local samconfig.toml file, an S3 bucket, and an ECR repository. When prompted for configuration info you can use the defaults the tool suggests

$ sam deploy --guided

For future deploys you can just run

$ sam deploy

Once the deploy has finished we can check that it is running by tailing the CloudWatch logs.

Use the stack-name you choose when you ran sam deploy –guided

$ sam logs -n ClojureAwsLambdaExampleFunction --stack-name sam-app --tail

After a few minutes of running you should see output similar to

2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:00:14.729000 START RequestId: cc9f8cfd-04bf-44af-9b88-8d0349adf74e Version
: $LATEST
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:00:14.734000 -handleRequest called! with event:  {:detail-type Scheduled Event, :time 2015-10-08T16:53:06Z, :source aws.events, :account 123456789012, :region us-east-1, :id 53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa, :version 0, :resources [arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule], :detail {}}
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:00:14.736000 END RequestId: cc9f8cfd-04bf-44af-9b88-8d0349adf74e
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:00:14.736000 REPORT RequestId: cc9f8cfd-04bf-44af-9b88-8d0349adf74e  Durat
ion: 6.05 ms    Billed Duration: 2637 ms        Memory Size: 256 MB     Max Memory Used: 101 MB Init Duration: 2630.22 ms
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:01:15.451000 START RequestId: 4dfa088e-8328-411e-bf34-e70fe0f74691 Version
: $LATEST
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:01:15.457000 -handleRequest called! with event:  {:detail-type Scheduled Event, :time 2015-10-08T16:53:06Z, :source aws.events, :account 123456789012, :region us-east-1, :id 53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa, :version 0, :resources [arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule], :detail {}}
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:01:15.475000 END RequestId: 4dfa088e-8328-411e-bf34-e70fe0f74691
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:01:15.475000 REPORT RequestId: 4dfa088e-8328-411e-bf34-e70fe0f74691  Durat
ion: 18.81 ms   Billed Duration: 19 ms  Memory Size: 256 MB     Max Memory Used: 102 MB
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:02:11.605000 START RequestId: 13ae2a9c-3d0d-40eb-8427-1e7807bff399 Version
: $LATEST
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:02:11.609000 -handleRequest called! with event:  {:detail-type Scheduled Event, :time 2015-10-08T16:53:06Z, :source aws.events, :account 123456789012, :region us-east-1, :id 53dc4d37-cffa-4f76-80c9-8b7d4a4d2eaa, :version 0, :resources [arn:aws:events:us-east-1:123456789012:rule/my-scheduled-rule], :detail {}}
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:02:11.609000 END RequestId: 13ae2a9c-3d0d-40eb-8427-1e7807bff399
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:02:11.609000 REPORT RequestId: 13ae2a9c-3d0d-40eb-8427-1e7807bff399  Durat
ion: 1.06 ms    Billed Duration: 2 ms   Memory Size: 256 MB     Max Memory Used: 102 MB
2021/10/17/[$LATEST]e4f5879df90c4d2f90bf036dfc1773c7 2021-10-17T16:03:11.363000 START RequestId: 76b568e8-8fa2-446f-803e-b9252db1b9a7 Version
: $LATEST

Clean up

When you are finished, you can delete the resources associated with the stack

Use the stack-name you choose when you ran sam deploy –guided

$ sam delete --stack-name sam-app

Cats vs Dogs - Part 3 - 99.1% Accuracy - Binary Image Classification with PyTorch and an Ensemble of ResNet Models

Sun, 12 Apr 2020 00:00:00 +0000

In 2014 Kaggle ran a competition to determine if images contained a dog or a cat. In this series of posts we'll see how easy it is to use Keras to create a 2D convolutional neural network that potentially could have won the contest.

In part 1 we used Keras to define a neural network architecture from scratch and were able to get to 92.8% categorization accuracy.

In part 2 we used once again used Keras and a VGG16 network with transfer learning to achieve 98.6% accuracy.

In this post we'll switch gears to use PyTorch with an ensemble of ResNet models to reach 99.1% accuracy.

This post was inspired by the book Programming PyTorch for Deep Learning by Ian Pointer.

Code is available in a jupyter notebook here. You will need to download the data from the Kaggle competition. The dataset contains 25,000 images of dogs and cats (12,500 from each class). We will create a new dataset containing 3 subsets, a training set with 16,000 images, a validation dataset with 4,500 images and a test set with 4,500 images.

Build the networks

import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torch.nn.functional as F
import torchvision
import torchvision.models as models
from torchvision import transforms
from PIL import Image
import matplotlib.pyplot as plt

Download models pretrained on ImageNet with PyTorch Hub

model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)
model_resnet34 = torch.hub.load('pytorch/vision', 'resnet34', pretrained=True)

Since we are doing transfer learning we want to freeze all params except the BatchNorm layers, as here they are trained to the mean and standard deviation of ImageNet and we may lose some signal.

for name, param in model_resnet18.named_parameters():
    if("bn" not in name):
        param.requires_grad = False

for name, param in model_resnet34.named_parameters():
    if("bn" not in name):
        param.requires_grad = False

Next we want to replace the classifier so we can make predictions on our dataset, rather than the 1,000 classes from ImageNet the model was trained on.

num_classes = 2

model_resnet18.fc = nn.Sequential(nn.Linear(model_resnet18.fc.in_features,512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

model_resnet34.fc = nn.Sequential(nn.Linear(model_resnet34.fc.in_features,512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

Functions for training and loading data

Create a function we can use to train the model.

def train(model, optimizer, loss_fn, train_loader, val_loader, epochs=5, device="cpu"):
    for epoch in range(epochs):
        training_loss = 0.0
        valid_loss = 0.0
        model.train()
        for batch in train_loader:
            optimizer.zero_grad()
            inputs, targets = batch
            inputs = inputs.to(device)
            targets = targets.to(device)
            output = model(inputs)
            loss = loss_fn(output, targets)
            loss.backward()
            optimizer.step()
            training_loss += loss.data.item() * inputs.size(0)
        training_loss /= len(train_loader.dataset)

        model.eval()
        num_correct = 0 
        num_examples = 0
        for batch in val_loader:
            inputs, targets = batch
            inputs = inputs.to(device)
            output = model(inputs)
            targets = targets.to(device)
            loss = loss_fn(output,targets) 
            valid_loss += loss.data.item() * inputs.size(0)

            correct = torch.eq(torch.max(F.softmax(output, dim=1), dim=1)[1], targets).view(-1)
            num_correct += torch.sum(correct).item()
            num_examples += correct.shape[0]
        valid_loss /= len(val_loader.dataset)

        print('Epoch: {}, Training Loss: {:.4f}, Validation Loss: {:.4f}, accuracy = {:.4f}'.format(epoch, training_loss,
        valid_loss, num_correct / num_examples))

Next create some code to load and process our training, test, and validation images.

batch_size=32
img_dimensions = 224

# Normalize to the ImageNet mean and standard deviation
# Could calculate it for the cats/dogs data set, but the ImageNet
# values give acceptable results here.
img_transforms = transforms.Compose([
    transforms.Resize((img_dimensions, img_dimensions)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225] )
    ])

img_test_transforms = transforms.Compose([
    transforms.Resize((img_dimensions,img_dimensions)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225] )
    ])

def check_image(path):
    try:
        im = Image.open(path)
        return True
    except:
        return False

train_data_path = "/home/wtf/dogs-vs-cats/train/"
train_data = torchvision.datasets.ImageFolder(root=train_data_path,transform=img_transforms, is_valid_file=check_image)

validation_data_path = "/home/wtf/dogs-vs-cats/validation/"
validation_data = torchvision.datasets.ImageFolder(root=validation_data_path,transform=img_test_transforms, is_valid_file=check_image)

test_data_path = "/home/wtf/dogs-vs-cats/test/"
test_data = torchvision.datasets.ImageFolder(root=test_data_path,transform=img_test_transforms, is_valid_file=check_image)

num_workers = 6
train_data_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=num_workers)
validation_data_loader = torch.utils.data.DataLoader(validation_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)
test_data_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)

if torch.cuda.is_available():
    device = torch.device("cuda") 
else:
    device = torch.device("cpu")

Lets verify that the numbers look correct

print(f'Num training images: {len(train_data_loader.dataset)}')
print(f'Num validation images: {len(validation_data_loader.dataset)}')
print(f'Num test images: {len(test_data_loader.dataset)}')

Which should output:

Num training images: 16000
Num validation images: 4500
Num test images: 4500

Train and test the models

def test_model(model):
    correct = 0
    total = 0
    with torch.no_grad():
        for data in test_data_loader:
            images, labels = data[0].to(device), data[1].to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    print('correct: {:d}  total: {:d}'.format(correct, total))
    print('accuracy = {:f}'.format(correct / total))

Train the ResNet18 model for a couple epochs. We could let it go longer (and use a larger batch size above), but I've been using a relatively ancient 6 year old GPU for this post, and not wanting to wait forever these settings are good enough for a blog post.

model_resnet18.to(device)
optimizer = optim.Adam(model_resnet18.parameters(), lr=0.001)
train(model_resnet18, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)

Epoch: 0, Training Loss: 0.0855, Validation Loss: 0.0358, accuracy = 0.9878
Epoch: 1, Training Loss: 0.0498, Validation Loss: 0.0309, accuracy = 0.9873

Now check against our holdout test set

test_model(model_resnet18)

correct: 4456  total: 4500
accuracy = 0.990222

And do the same for our ResNet34 network

model_resnet34.to(device)
optimizer = optim.Adam(model_resnet34.parameters(), lr=0.001)
train(model_resnet34, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)

Epoch: 0, Training Loss: 0.0678, Validation Loss: 0.0239, accuracy = 0.9907
Epoch: 1, Training Loss: 0.0354, Validation Loss: 0.0317, accuracy = 0.9887

And test

test_model(model_resnet34)

correct: 4450  total: 4500
accuracy = 0.988889

This gives us two models, one with 99.0% accuracy on our test set and 98.9% on the other.

Make some predictions

Lets check a couple individual images from the test set.

import os
def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    return classes, class_to_idx

def make_prediction(model, filename):
    labels, _ = find_classes('/home/wtf/dogs-vs-cats/test')
    img = Image.open(filename)
    img = img_test_transforms(img)
    img = img.unsqueeze(0)
    prediction = model(img.to(device))
    prediction = prediction.argmax()
    print(labels[prediction])

make_prediction(model_resnet34, '/home/wtf/dogs-vs-cats/test/dogs/dog.11460.jpg')
make_prediction(model_resnet34, '/home/wtf/dogs-vs-cats/test/cats/cat.12262.jpg')

Which outputs:

dogs
cats

Seems reasonable.

Save and load models

Since we don't want to have to train the models again every time we start up a jupyter notebook, lets see how we can save them to disk and then reload them.

torch.save(model_resnet18.state_dict(), "./model_resnet18.pth")
torch.save(model_resnet34.state_dict(), "./model_resnet34.pth")

# Remember that you must call model.eval() to set dropout and batch normalization layers to
# evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

resnet18 = torch.hub.load('pytorch/vision', 'resnet18')
resnet18.fc = nn.Sequential(nn.Linear(resnet18.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet18.load_state_dict(torch.load('./model_resnet18.pth'))
resnet18.eval()

resnet34 = torch.hub.load('pytorch/vision', 'resnet34')
resnet34.fc = nn.Sequential(nn.Linear(resnet34.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet34.load_state_dict(torch.load('./model_resnet34.pth'))
resnet34.eval()

Test with an ensemble

We'll use a very simple ensemble here. Take the prediction for each image from each model, average them to generate a new prediction for the image.

# Test against the average of each prediction from the two models
models_ensemble = [resnet18.to(device), resnet34.to(device)]
correct = 0
total = 0
with torch.no_grad():
    for data in test_data_loader:
        images, labels = data[0].to(device), data[1].to(device)
        predictions = [i(images).data for i in models_ensemble]
        avg_predictions = torch.mean(torch.stack(predictions), dim=0)
        _, predicted = torch.max(avg_predictions, 1)

        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('accuracy = {:f}'.format(correct / total))
print('correct: {:d}  total: {:d}'.format(correct, total))

Which results in

accuracy = 0.990889
correct: 4459  total: 4500

The magic of ensembles is that given two models with accuracy of 0.990222 and 0.988889 we are able to make predictions and get to 0.990889, which is higher than any individual model.

In this case we aren't seeing a dramatic increase, but ensembles can be very useful. I once had an entry in a Kaggle competition with around 4,000 entrants where my best individual model put me in the top 10%, but by combining a number of entries into an ensemble placed me in the top 2%.

Next steps

There is a lot we didn't do here. You could try augmenting the training images with TorchVision, try different ways of creating the ensemble, add a model using a different network like VGG from TorchHub to the ensemble, etc.

CHIP-8 emulator in WebAssembly using Rust

Sun, 26 Jan 2020 00:00:00 +0000

Lately I have been working on an emulator for the CHIP-8 that can compile to WebAssembly using Rust. The source code is available on GitHub. And you can play a game of pong here.

I've always wanted to write an emulator for an older hardware like the Nintendo Entertainment System or Game Boy Advance, but decided to start with a simpler project that could be finished over the course of a few nights. The CHIP-8 is a great starter emulation project, it has:

A relatively small instruction set (35 instructions)
16 general purpose registers
A 64x32 monochromatic display
A 16 character hex based keypad
No interrupts, but it does have two timer registers that count down at 60 Hz

A few references I found useful to get started are this post by Laurence Muller, Cowgod's Chip-8 Technical Reference, and this post by Colin Eberhardt. If you want to create your own emulator I highly recommend taking a look at the above pages.

Implementation

I won't go too deep into implementations details here, but if you want to know more the source code is available. I represented the CPU in Rust like this:

pub struct Cpu {
    // 4k Memory
    pub memory: [u8; 4096],
    // Program Counter
    pub pc: u16,
    // 16 general purpose 8-bit registers, usually referred to as Vx, where x is a hexadecimal digit (0 through F)
    pub v: [u8; 16],
    // Index register
    pub i: u16,
    // The stack is an array of 16 16-bit values
    pub stack: [u16; 16],
    // Stack pointer
    pub sp: u8,
    // Display - 64x32 pixels
    pub display: [u8; 64 * 32],
    // Delay timer
    pub dt: u8,
    // Sound timer
    pub st: u8,
    // Keyboard
    pub keys: [bool; 16],
}

Memory

The system's memory map looks like:

0x000-0x1FF - In original implementations the CHIP 8 interpreter.
0x050-0x0A0 - In emulators stores the 4x5 pixel font set (0-F).
0x200-0xFFF - Program ROM and work RAM.

One of the first things you will need to do is load the program into memory starting at position 0x200.

Opcodes

The CHIP-8 has 35 opcodes you will need to handle, all of which are two bytes long. First you need to extract the opcode, since memory elements are 8 bits and an opcode is 16 bits, we need to fetch two locations from memory and do some bit shifting to get a u16 opcode.

let code1: u16 = self.memory[self.pc as usize] as u16;
let code2: u16 = self.memory[(self.pc + 1) as usize] as u16;
let opcode: u16 = code1 << 8 | code2;

An opcode will look something like 0x7301. I highly recommend writing a disassembler to make it easier to read. An example of doing so is here. In this case the 0x7301 is displayed as ADD V3, 01 or in other words, add 1 to the current value of register V3.

Thanks to Rust's support for pattern matching over ranges, handling each opcode is relatively easy to do:

match opcode {
    0x7000..= 0x7FFF => {
        // 7xkk - ADD Vx, byte
        // Set Vx = Vx + kk.
        let x = ((opcode & 0x0F00) >> 8) as usize;
        let kk = (opcode & 0x00FF) as u8;

        // Note the overflowing_add, since addition values are 8 bits adding
        // 255 + 2 should overflow and result in 1.
        let (result, _) = self.v[x].overflowing_add(kk);
        self.v[x] = result;
        // Increment the program counter to point at the next opcode.
        self.pc += 2;
    },

    // ...
    // code to handle the remaining opcodes would go here
    // ...

    _ => {
        // This is an opcode we don't know how to handle yet
        self.pc += 2;
        // EmulateCycleError is a custom error defined elsewhere in the code
        let error = EmulateCycleError { message: format!("{:X} opcode not handled", opcode) };
        return Err(error);
    }
}

Handling keyboard input in JavaScript

Running the emulator and communicating between JavaScript and WebAssembly is easy thanks to the Rust wasm-pack tool.

In this case we annotate this Rust code with #[wasm_bindgen] so we can call it from JavaScript. Note that we are using the unsafe keyword here because CPU is a static value that we hold it's state in.

#[wasm_bindgen]
pub fn key_down(key: u8) {
    unsafe {
        CPU.keys[key as usize] = true;
    }
}

Then in JavaScript we can call update the state of the keypad on the WebAssembly side like this:

import("./crate/pkg/index.js").then(wasm => {
  document.addEventListener("keydown", event => {
    let keyCode = keyMap[event.key];
    if (keyCode >= 0 && keyCode <= 0xf) {
      wasm.key_down(keyMap[event.key]);
    }
  });
});

The keyMap is a mapping from the user's keyboard to the hex based keypad on the CHIP-8, and looks like this:

// CHIP-8 Keypad    User Keyboard
// +-+-+-+-+        +-+-+-+-+
// |1|2|3|C|        |1|2|3|4|
// +-+-+-+-+        +-+-+-+-+
// |4|5|6|D|        |Q|W|E|R|
// +-+-+-+-+   <=   +-+-+-+-+
// |7|8|9|E|        |A|S|D|F|
// +-+-+-+-+        +-+-+-+-+
// |A|0|B|F|        |Z|X|C|V|
// +-+-+-+-+        +-+-+-+-+

const keyMap = {
  '1': 0x1,
  '2': 0x2,
  '3': 0x3,
  '4': 0xc,

  'q': 0x4,
  'w': 0x5,
  'e': 0x6,
  'r': 0xd,

  'a': 0x7,
  's': 0x8,
  'd': 0x9,
  'f': 0xe,

  'z': 0xa,
  'x': 0x0,
  'c': 0xb,
  'v': 0xf,
};

Running the emulator

We use window.requestAnimationFrame() to run 9 cycles on the cpu 60 times per second, then update the HTML canvas display and the UI. We run 9 cycles because the CHIP-8 seems to run at about 500hz, and calling 9 cycles 60 times per second gets us to 540 hZ, which is close enough for this project.

Note that with window.requestAnimationFrame()

The number of callbacks is usually 60 times per second
but will generally match the display refresh rate in most web
browsers as per W3C recommendation.

So if your monitor's refresh rate is not 60 Hz the emulator will likely run either too fast or slow. For this project it is a compromise i'm ok with, but it is something that could be improved.

Ray Tracing with Rust and WebAssembly

Fri, 17 Jan 2020 00:00:00 +0000

I have been working through the book The Ray Tracer Challenge: A Test-Driven Guide to Your First 3D Renderer by Jamis Buck. It is a fantastic book, there is almost no code (other than a little bit of pseudocode), instead there is an explanation of how to build a Whitted ray tracer and a large number of unit tests to walk you through implementing it yourself.

I decided to try to implement the ray tracer in Rust and compile to both a native app and WebAssembly. Source code is available at GitHub. Currently i've worked through chapter 9 (of 17) so there are a number of features my implementation is missing (and I haven't done any optimization yet).

Unsurprisingly the native app is much faster (by about an order of magnitude by my very unscientific measurements), but I was quite pleased how easy it was to generate the WebAssembly code using Rust.

Above is a pregenerated image, but you can click the button below to generate a smaller lower quality picture on demand. I am using Web Workers to do the rendering calculations in multiple threads (each Web Worker runs some WebAssembly).

If you are using Safari there is an issue with window.navigator.hardwareConcurrency and this app will default to a single thread. Try in a browser like Firefox or Chrome to see better performance.

I have been quite impressed with the Rust tooling and how easy it was to generate a WebAssembly app in the browser. I'm now intrigued with the idea of being able to write WebAssembly modules in languages like C/C++/Rust/Go etc which can interop with each other, and then have the ability to write a smallish frontend in JavaScript or Python (using something like the Wasmtime package) to glue it all together.

Cats vs Dogs - Part 2 - 98.6% Accuracy - Binary Image Classification with Keras and Transfer Learning

Sun, 12 May 2019 00:00:00 +0000

In this post we'll see how we can fine tune a network pretrained on ImageNet and take advantage of transfer learning to reach 98.6% accuracy (the winning entry scored 98.9%).

In part 1 we used Keras to define a neural network architecture from scratch and were able to get to 92.8% categorization accuracy.

In part 3 we'll switch gears a bit and use PyTorch instead of Keras to create an ensemble of models that provides more predictive power than any single model and reaches 99.1% accuracy.

The code is available in a jupyter notebook here. You will need to download the data from the Kaggle competition. The dataset contains 25,000 images of dogs and cats (12,500 from each class). We will create a new dataset containing 3 subsets, a training set with 16,000 images, a validation dataset with 4,500 images and a test set with 4,500 images.

VGG16

We'll use the VGG16 architecture as described in the paper Very Deep Convolutional Networks for Large-Scale Image Recognition by Karen Simonyan and Andrew Zisserman. We're using it because it has a relatively simple architecture and Keras ships with a model that has been pretrained on ImageNet.

Keras includes a number of additional pretrained networks if you want to try with a different one. You could also build the VGG16 network yourself, the code for the Keras implementation is here. It is just a number of Conv2D and MaxPooling2D layers with a dense network on top with a final softmax activation function.

Predict what an image contains using VGG16

First we'll make predictions on what one of our images contained. The Keras VGG16 model provided was trained on the ILSVRC ImageNet images containing 1,000 categories. It will be especially useful in this case since it 90 of the 1,000 categories are species of dogs.

First lets take a peek at an image.

from keras.preprocessing import image
from matplotlib.pyplot import imshow

fnames = [os.path.join(train_dogs_dir, fname) for fname in os.listdir(train_dogs_dir)]
img_path = fnames[1] # Choose one image to view
img = image.load_img(img_path, target_size=(224, 224)) # load image and resize it
x = image.img_to_array(img) # Convert to a Numpy array with shape (224, 224, 3)

x = x.reshape((1,) + x.shape)

plt.imshow(image.array_to_img(x[0]))

Now lets ask the model what it thinks the picture is.

from keras.applications.imagenet_utils import decode_predictions
from keras.applications import VGG16

model = VGG16(weights='imagenet', include_top=True)

features = model.predict(x)
decode_predictions(features, top=5)

[[('n02097298', 'Scotch_terrier', 0.84078884),
  ('n02105412', 'kelpie', 0.07755529),
  ('n02105056', 'groenendael', 0.048816346),
  ('n02106662', 'German_shepherd', 0.006882491),
  ('n02104365', 'schipperke', 0.005642254)]]

It thinks there is an 84% chance it's a Scotch Terrier, and the other top predictions are all dogs. Seems pretty reasonable.

Train a Cats vs Dogs classifier

We can also ask Keras to provide us with the model trained on ImageNet, but without the top dense layers. Then it is just a matter of adding our own dense layers (note that since we are doing binary classification we've used a sigmoid activation function in the final layer). And tell the model to only train the dense layers we created (we don't want to retrain the lower layers that have learnt from ImageNet).

In this case it works so well because ImageNet has a large number of animal pictures, so the lower layers already have a sort of conception of what is "dogness" and "catness".

from keras import layers, models, optimizers

conv_base = VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(224, 224, 3))

model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

conv_base.trainable = False

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

Lets create a generator for the images

from keras.applications.vgg16 import preprocess_input
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

# The list of classes will be automatically inferred from the subdirectory names/structure under train_dir
train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(224, 224), # resize all images to 224 x 224
    batch_size=50,
    class_mode='binary') # because we use binary_crossentropy loss we need binary labels

validation_generator = test_datagen.flow_from_directory(
    validation_dir,
    target_size=(224, 224), # resize all images to 224 x 224
    batch_size=50,
    class_mode='binary')

Found 16000 images belonging to 2 classes.
Found 4500 images belonging to 2 classes.

And train.

history = model.fit_generator(
    train_generator,
    steps_per_epoch=320, # batches in the generator are 50, so it takes 320 batches to get to 16000 images
    epochs=30,
    validation_data=validation_generator,
    validation_steps=90) # batches in the generator are 50, so it takes 90 batches to get to 4500 images

Epoch 1/30
320/320 [==============================] - 139s 434ms/step - loss: 0.7365 - acc: 0.9238 - val_loss: 0.2217 - val_acc: 0.9751
Epoch 2/30
320/320 [==============================] - 137s 428ms/step - loss: 0.2950 - acc: 0.9689 - val_loss: 0.2579 - val_acc: 0.9736
...
...
...
Epoch 29/30
320/320 [==============================] - 137s 429ms/step - loss: 0.0206 - acc: 0.9977 - val_loss: 0.2079 - val_acc: 0.9818
Epoch 30/30
320/320 [==============================] - 137s 429ms/step - loss: 0.0146 - acc: 0.9982 - val_loss: 0.2063 - val_acc: 0.9824

Lets take a look at what accuracy and loss looked like during training. The code to generate the plot is available in the posts jupyter notebook here.

We seem to be overfitting (and probably could have trained for far fewer than 30 epochs), but the results are still pretty good. Lets compare against the holdout test set.

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

test_loss, test_acc = model.evaluate_generator(test_generator, steps=90)
print('test acc:', test_acc)

Found 4500 images belonging to 2 classes.
test acc: 0.9833333353201549

A validation accuracy of 98.2% and test of 98.3%. Not bad, but lets see if we can do better.

Transfer learning / Fine tune model

We can also retrain the later convolutional layers. Here is how we can do it with VGG16.

conv_base = VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(224, 224, 3))

conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-5),
              metrics=['acc'])

And lets fit the model

history = model.fit_generator(
    train_generator,
    steps_per_epoch=320, # batches in the generator are 50, so it takes 320 batches to get to 16000 images
    epochs=30,
    validation_data=validation_generator,
    validation_steps=90) # batches in the generator are 50, so it takes 90 batches to get to 4500 images

Epoch 1/30
320/320 [==============================] - 151s 471ms/step - loss: 0.7004 - acc: 0.9150 - val_loss: 0.1462 - val_acc: 0.9720
Epoch 2/30
320/320 [==============================] - 150s 469ms/step - loss: 0.1335 - acc: 0.9716 - val_loss: 0.0935 - val_acc: 0.9784
...
...
...
Epoch 29/30
320/320 [==============================] - 151s 470ms/step - loss: 0.0035 - acc: 0.9998 - val_loss: 0.1489 - val_acc: 0.9858
Epoch 30/30
320/320 [==============================] - 151s 470ms/step - loss: 0.0040 - acc: 0.9996 - val_loss: 0.1471 - val_acc: 0.9867

Lets take a look at accuracy and loss again.

We still seem to be overfitting, suggesting we could potentially improve performance. The params on the top dense layers were fairly arbitrarily for this post. I'll leave improving performance further as an exercise for the reader. We've also trained on just 16,000 of the 25,000 images available. If the contest was still going we could retrain with the additional 9k images and likely have better results.

Finally lets compare to the test set.

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

test_loss, test_acc = model.evaluate_generator(test_generator, steps=90)
print('test acc:', test_acc)

Found 4500 images belonging to 2 classes.
test acc: 0.9862222280767229

98.6% accuracy. Not bad at all, it goes to show how lucky we have it these days. This would have been competitive with a state of the art solution not that many years ago, and now we can achieve it with Keras and just a few lines of Python.

Cats vs Dogs - Part 1 - 92.8% Accuracy - Binary Image Classification with Keras and Deep Learning

Tue, 07 May 2019 00:00:00 +0000

We will start with a basic neural network that is 84% accurate at predicting whether an image contains a cat or dog. Then we'll add dropout and finally add data augmentation to get to 92.8% categorization accuracy.

In part 2 we'll see how we can fine tune a network pretrained on ImageNet and take advantage of transfer learning to reach 98.6% accuracy (the winning entry scored 98.9%).

In part 3 we'll switch gears and use PyTorch instead of Keras to create an ensemble of models that provides more predictive power than any single model and reaches 99.1% accuracy.

Build the first network

We'll use the Keras sequential API to build a pretty straightforward network that consists of a number of Conv2D and pooling layers with a dense network at the end that makes the cat or dog prediction.

from keras import layers, models, optimizers

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-4),
              metrics=['acc'])

We'll create some generators that will do some preprocessing on the images and resize them to be 224x224 pixels.

from keras.preprocessing.image import ImageDataGenerator

# Rescale pixel values from [0, 255] to [0, 1]
train_datagen = ImageDataGenerator(rescale=1./255) 
test_datagen = ImageDataGenerator(rescale=1./255)

# The list of classes will be automatically inferred from the subdirectory names/structure under train_dir
train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(224, 224), # resize all images to 224 x 224
    batch_size=50,
    class_mode='binary') # because we use binary_crossentropy loss we need binary labels

validation_generator = test_datagen.flow_from_directory(
    validation_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

Found 16000 images belonging to 2 classes.
Found 4500 images belonging to 2 classes.

Now lets fit the network for 30 epochs.

history = model.fit_generator(
    train_generator,
    steps_per_epoch=320, # 50 batches in the generator, so it takes 320 batches to get to 16000 images
    epochs=30,
    validation_data=validation_generator,
    validation_steps=90) # 90 x 50 == 4500

Epoch 1/30
320/320 [==============================] - 48s 150ms/step - loss: 0.6070 - acc: 0.6579 - val_loss: 0.5263 - val_acc: 0.7413
Epoch 2/30
320/320 [==============================] - 45s 142ms/step - loss: 0.5145 - acc: 0.7433 - val_loss: 0.5812 - val_acc: 0.6971
...
...
...
Epoch 29/30
320/320 [==============================] - 46s 143ms/step - loss: 0.0133 - acc: 0.9959 - val_loss: 0.9195 - val_acc: 0.8349
Epoch 30/30
320/320 [==============================] - 46s 143ms/step - loss: 0.0133 - acc: 0.9965 - val_loss: 0.8704 - val_acc: 0.8351

Plotting the results (code to generate the graphs is available in the jupyter notebook) we can see that the model begins to overfit the training data almost immediately. The training accuracy continues to improve until it gets close to 100%, while the validation set tops out at around 84% and loss degrades as training continues.

In the final epoch we reached 83.5% accuracy on the validation set, lets quickly double check on the test set to compare.

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

test_loss, test_acc = model.evaluate_generator(test_generator, steps=90)
print('test acc:', test_acc)

Found 4500 images belonging to 2 classes.
test acc: 0.8406666629844242

On one hand this isn't horrible, a baseline where we pick dog every time would have 50% accuracy. But on the other hand we can do much better.

Add Dropout

We can add dropout to regularize the model. Adding dropout randomly removes a percentage of neurons (sets their value to 0) at each update during training time, which helps prevent overfitting. By randomly throwing out data this will help to prevent the network from becoming too smart and essentially memorizing the training data, and thereby generalize to new data better.

from keras import layers, models, optimizers

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5)) # Note the only change is that we added dropout here
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-4),
              metrics=['acc'])

How does this model perform?

Epoch 1/30
320/320 [==============================] - 46s 145ms/step - loss: 0.6166 - acc: 0.6449 - val_loss: 0.5348 - val_acc: 0.7378
Epoch 2/30
320/320 [==============================] - 46s 143ms/step - loss: 0.5280 - acc: 0.7364 - val_loss: 0.4777 - val_acc: 0.7733
...
...
...
Epoch 29/30
320/320 [==============================] - 46s 144ms/step - loss: 0.1037 - acc: 0.9603 - val_loss: 0.3722 - val_acc: 0.8720
Epoch 30/30
320/320 [==============================] - 46s 144ms/step - loss: 0.1030 - acc: 0.9604 - val_loss: 0.3444 - val_acc: 0.8780

We still seem to be overfitting, but not as badly. We've also increase validation accuracy to 87.8%, so this is a bit of a win. We could try tuning the network architecture or the dropout amount, but instead lets try something else next.

Add Data Augmentation

Keras includes an ImageDataGenerator class which lets us generate a number of random transformations on an image. Since we're only training on 16,000 images, we can use this to create "new" images to help the network learn. There are limits to how much this can help, but in this case we will get a decent accuracy boost.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')

The above creates a data generator that will take in an image from the training set and return an image that has been slightly altered. Lets visualize what this looks like.

from keras.preprocessing import image

fnames = [os.path.join(train_cats_dir, fname) for fname in os.listdir(train_cats_dir)]
img_path = fnames[4] # Choose one image to augment
img = image.load_img(img_path, target_size=(224, 224)) # load image and resize it
x = image.img_to_array(img) # Convert to a Numpy array with shape (224, 224, 3)
x = x.reshape((1,) + x.shape)

# Generates batches of randomly transformed images.
# Loops indefinitely, so you need to break once three images have been created
i = 0
for batch in datagen.flow(x, batch_size=1):
    plt.figure(i)
    imgplot = plt.imshow(image.array_to_img(batch[0]))
    i += 1
    if i % 3 == 0:
        break
plt.show()

We'll use the same network architecture as the previous step, but generate images like this

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255) # Note that validation data should not be augmented

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

Found 16000 images belonging to 2 classes.
Found 4500 images belonging to 2 classes.

And then increase the steps per epoch 3x so that we are training on 48,000 images instead of 16,000. This takes awhile as in the example I'm running on a 5 year old GPU I had laying around the house, training should be much faster if you can use more recent hardware or rent time on a GPU in the cloud.

history = model.fit_generator(
    train_generator,
    steps_per_epoch=960, # 960 x 50 = 48000 (we are showing different augmented images more than once per epoch)
    epochs=30,
    validation_data=validation_generator,
    validation_steps=90) # 90 x 50 == 4500

Epoch 1/30
960/960 [==============================] - 411s 429ms/step - loss: 0.6202 - acc: 0.6428 - val_loss: 0.5285 - val_acc: 0.7413
Epoch 2/30
960/960 [==============================] - 410s 427ms/step - loss: 0.5495 - acc: 0.7141 - val_loss: 0.5082 - val_acc: 0.7447
...
...
...
Epoch 29/30
960/960 [==============================] - 409s 426ms/step - loss: 0.2323 - acc: 0.9057 - val_loss: 0.2284 - val_acc: 0.9136
Epoch 30/30
960/960 [==============================] - 413s 430ms/step - loss: 0.2330 - acc: 0.9040 - val_loss: 0.1821 - val_acc: 0.9222

And finally compare to the holdout test set.

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size=(224, 224),
    batch_size=50,
    class_mode='binary')

test_loss, test_acc = model.evaluate_generator(test_generator, steps=90)
print('test acc:', test_acc)

Found 4500 images belonging to 2 classes.
test acc: 0.928444442484114

Yet another improvement, we've reached 92.8% on the test set, are seeing similar numbers on the validation data, and overfitting doesn't seem to be nearly as much of a problem.

We could work to improve these results, but that's all for today. In the next post we'll see how we can take advantage of transfer learning to reach 98.6% accuracy.

MNIST Image Classification using Deep Learning and Keras

Sun, 21 Apr 2019 00:00:00 +0000

In this post we'll use Keras to build the hello world of machine learning, classify a number in an image from the MNIST database of handwritten digits, and achieve ~99% classification accuracy using a convolutional neural network.

Much of this is inspired by the book Deep Learning with Python by François Chollet. I highly recommend reading the book if you would like to dig deeper or learn more.

If you would like to follow along, the code is also available in a jupyter notebook here.

from keras import models, layers
from keras.datasets import mnist
from keras.utils import to_categorical
import matplotlib.pyplot as plt
import numpy as np

Since working with the MNIST digits is so common, Keras provides a function to load the data. You can see a full list of datasets Keras has packaged up.

Let's load the data:

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

The training set consists of 60,000 28x28 pixel images, and the test set 10,000.

train_images.shape, test_images.shape

((60000, 28, 28), (10000, 28, 28))

Lets look at the first ten training images. They are each 28x28 grayscale images with one color value between 0 and 255.

_, ax = plt.subplots(1, 10, figsize=(10,10))

for i in range(0, 10):
    ax[i].axis('off')
    ax[i].imshow(train_images[i], cmap=plt.cm.binary)

And the labels representing which class the image represents.

train_labels[0:10]

array([5, 0, 4, 1, 9, 2, 1, 3, 1, 4], dtype=uint8)

Build the neural network

Now build the neural network. We'll be using a number of convolutional layers. Note that we only have to specify the input shape in the first layer. The last layer provides the output. It has 10 units (one for each digit 0 to 9) and uses a softmax activation to map the output of a network to a probability distribution over the predicted output classes.

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

One way to see what the network looks like is to use the summary() function:

model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64)          0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 3, 3, 64)          36928     
_________________________________________________________________
flatten_1 (Flatten)          (None, 576)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                36928     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                650       
=================================================================
Total params: 93,322
Trainable params: 93,322
Non-trainable params: 0
_________________________________________________________________

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

We need to do some preprocessing of the images. We'll also use the first 50,000 training images for training, and the remaining 10,000 training examples for cross validation.

train_images = train_images.reshape((60000, 28, 28, 1))
train_images= train_images.astype('float32') / 255 # rescale pixel values from range [0, 255] to [0, 1]

test_images = test_images.reshape((10000, 28, 28, 1))
test_images= test_images.astype('float32') / 255

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

validation_images = train_images[50000:]
validation_labels = train_labels[50000:]

train_images = train_images[:50000]
train_labels = train_labels[:50000]

history = model.fit(train_images, train_labels, epochs=5, batch_size=64, validation_data=(validation_images, validation_labels))

Train on 50000 samples, validate on 10000 samples
Epoch 1/5
50000/50000 [==============================] - 20s 391us/step - loss: 0.1959 - acc: 0.9387 - val_loss: 0.0798 - val_acc: 0.9760
Epoch 2/5
50000/50000 [==============================] - 19s 380us/step - loss: 0.0509 - acc: 0.9845 - val_loss: 0.0513 - val_acc: 0.9849
Epoch 3/5
50000/50000 [==============================] - 19s 382us/step - loss: 0.0343 - acc: 0.9892 - val_loss: 0.0408 - val_acc: 0.9880
Epoch 4/5
50000/50000 [==============================] - 19s 379us/step - loss: 0.0257 - acc: 0.9918 - val_loss: 0.0448 - val_acc: 0.9874
Epoch 5/5
50000/50000 [==============================] - 19s 377us/step - loss: 0.0208 - acc: 0.9938 - val_loss: 0.0356 - val_acc: 0.9903

test_loss, test_acc = model.evaluate(test_images, test_labels)
print('Accuracy:', test_acc)
print('Loss: ', test_loss)

10000/10000 [==============================] - 1s 122us/step
Accuracy: 0.992
Loss:  0.027386048025220953

Looks pretty good we're seeing ~99% accuracy on the test set.

Visualize training

Now lets create a function that lets us graph the accuracy and loss values during training.

def plot_accuracy_and_loss(history):
    acc = history.history['acc']
    val_acc = history.history['val_acc']
    loss = history.history['loss']
    val_loss = history.history['val_loss']

    epochs = range(1, len(acc) + 1)

    plt.plot(epochs, acc, 'bo', label='Training acc')
    plt.plot(epochs, val_acc, 'b', label='Validation acc')
    plt.title('Training and validation accuracy')
    plt.legend()
    plt.show()

    plt.plot(epochs, loss, 'bo', label='Training loss')
    plt.plot(epochs, val_loss, 'b', label='Validation loss')
    plt.title('Training and validation loss')
    plt.legend()
    plt.show()

plot_accuracy_and_loss(history)

The above looks pretty good, we appear to be starting to overfit the data as we get further in, but training and validation sets are pretty close to each other.

Now lets look at a prediction, first we'll generate predictions for the test set.

preds = model.predict(test_images)

We'll use the network to try to figure out what the first digit in the test set is. If we manually look, it appears to be a 7.

# reload the test images so it will be in a format imshow() will understand
(_, _), (test_images, _) = mnist.load_data()

plt.imshow(test_images[0], cmap=plt.cm.binary)

Since the output of the network was a layer with 10 units and a softmax activation, we will get an array of length 10 with a prediction for each potential number. Here you can see that the network is 99.9% certain it is a seven.

print(preds[0])

[2.6081236e-12 1.8943378e-09 1.0174886e-08 6.8640638e-08 2.3309353e-11
 1.9539477e-10 7.4824168e-19 9.9999988e-01 4.3342949e-10 8.6599723e-09]

We can also find the class with the highest prediction score with a numpy function:

np.argmax(preds[0])

The next step would be to retrain the model with all 60,000 training examples (remember that in the model above we trained on 50,000 examples and validated on the remaining 10,000). I'll leave that as an exercise to the reader.

Ray Tracing in the Browser using ReasonML

Fri, 23 Nov 2018 00:00:00 +0000

I was reading Peter Shirley's little book Ray Tracing in One Weekend which provides code in C and for fun wanted to try to implement it in Reason. Source code is available here.

Above is a pregenerated image, but you can click the button below to generate a smaller lower quality picture on demand (it takes about 5 seconds to generate on my laptop).

I can't think of many reasons you'd want to do ray tracing in JavaScript, but for now this was a fun little weekend project.

Drawing Tilemaps on an HTML Canvas with ReasonML

Sun, 17 Jun 2018 00:00:00 +0000

I was curious about using Reason to draw 2D tilemaps in the browser using a Canvas element. If you need an overview of tiles, tilemaps, etc. there is a great introduction over at developer.mozilla.org.

We will end up drawing this level of what could be a roguelike game, with an adventurer, monsters, and treasure chests:

A full project with the source code is available here.

Sprites

We will be using a sprite atlas modified from this Angband tileset with a permissive license. It looks like this:

The level data is stored in a list, each number represents the index of the sprite atlas to use when drawing that tile.

let tiles = [ 
     2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
     2, 1, 1, 1, 1, 2, 5, 1, 1, 1, 1, 1, 1, 1, 1, 2, 
     2, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 
     2, 1, 1, 4, 1, 2, 2, 2, 2, 2, 0, 0, 0, 2, 1, 2,
     2, 1, 1, 1, 1, 2, 0, 0, 0, 0, 0, 0, 0, 2, 1, 2, 
     2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 
     2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 
     0, 2, 1, 1, 1, 2, 1, 2, 2, 1, 1, 1, 1, 1, 1, 2, 
     0, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 3, 1, 1, 2, 
     0, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 2, 
     0, 2, 1, 3, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 
     0, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 
     0, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 2, 
     0, 0, 2, 1, 1, 3, 1, 1, 2, 1, 1, 1, 1, 5, 1, 2, 
     0, 0, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 2, 
     0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 
   ];

Working with the DOM

Reason (and BuckleScript) allows us to write type safe code to work with the browser DOM, but we will need to write some code for the interop. You could just use bs.raw and use JavaScript directly, but doing it this way will provide some type safety and let you catch some errors at compile time instead of runtime.

Note: If you need to work with the DOM you should use the excellent bs-webapi library instead. The code below is just something I used for learning Reason/JavaScript interop.

module Window = {
  type t;
  [@bs.val] external t : t = "window";

  [@bs.send]
  external addEventListener : (t, string, unit => unit) => unit =
    "addEventListener";
};

module Document = {
  type t;
  [@bs.val] external t : t = "document";
};

module HtmlImageElement = {
  type t;
  [@bs.new] external make : unit => t = "Image";

  [@bs.set] external setSrc : (t, string) => unit = "src";

  [@bs.send]
  external addEventListener : (t, string, unit => unit) => unit =
    "addEventListener";

  /* Create a html  element, and return a promise that resolves when the */
  /* image has finished loading. */
  let loadFromSrc = imageSrc => {
    let imageEl = make();

    let loadImagePromise =
      Js.Promise.make((~resolve, ~reject) => {
        addEventListener(imageEl, "load", () => resolve(. imageEl));
        addEventListener(imageEl, "error", () =>
          reject(. Invalid_argument("Could not load image: " ++ imageSrc))
        );
      });

    setSrc(imageEl, imageSrc);
    loadImagePromise;
  };
};

module Canvas = {
  type t;
  [@bs.send]
  external getElementById : (Document.t, string) => t = "getElementById";
};

module Context = {
  type t;

  /* JavaScript equivalent: canvas.getContext('2d'); */
  [@bs.send]
  external getContext2d : (Canvas.t, [@bs.as "2d"] _) => t = "getContext";

  [@bs.send]
  external drawImage :
    (
      t,
      ~image: HtmlImageElement.t,
      ~dx: int,
      ~dy: int,
      ~dWidth: int,
      ~dHeight: int,
      ~sx: int,
      ~sy: int,
      ~sWidth: int,
      ~sHeight: int
    ) =>
    unit =
    "drawImage";
};

SpriteAtlas

The sprite atlas is an image with all the sprites in it, it looks like:

This code allows us to load the atlas into memory and draw one of the sprites into the context.

module SpriteAtlas = {
  type t = {
    imageElement: HtmlImageElement.t,
    spriteWidth: int,
  };

  let make = (src: string, spriteWidth: int) =>
    HtmlImageElement.loadFromSrc(src)
    |> Js.Promise.then_(imageElement =>
         Js.Promise.resolve({imageElement, spriteWidth})
       );

  let drawSprite = (~atlas: t, ~ctx, ~atlasNumber, ~x, ~y) =>
    Context.drawImage(
      ctx,
      ~image=atlas.imageElement,
      ~dx=atlasNumber * atlas.spriteWidth,
      ~dy=0, /*  In this code we only support atlases with a single row of images */
      ~dWidth=atlas.spriteWidth,
      ~dHeight=atlas.spriteWidth,
      ~sx=x,
      ~sy=y,
      ~sWidth=atlas.spriteWidth,
      ~sHeight=atlas.spriteWidth,
    );
};

Tilemap

The tilemap describes what our level looks like, it has a reference to a sprite atlas and a number of rows and columns representing what to draw in the level.

module TileMap = {
  type t = {
    atlas: SpriteAtlas.t,
    numRows: int,
    numCols: int,
    tileSize: int, /* pixels */
    tiles: list(int) /* Maybe should be an array so it is mutable? */
  };

  let make = (~atlas, ~numRows, ~numCols, ~tileSize, ~tiles) => {
    atlas,
    numRows,
    numCols,
    tileSize,
    tiles,
  };

  /* Given an index in a tilemap, get the row and column as if it was a 2D array */
  let getRowAndColumn = (idx: int, tilemap: t) : (int, int) => {
    let col: int = idx mod tilemap.numCols;
    let row: int =
      Js.Math.floor(float_of_int(idx) /. float_of_int(tilemap.numCols));
    (row, col);
  };

  let render = (ctx: Context.t, tilemap: t) =>
    List.iteri(
      (i, num) => {
        let (row, col) = getRowAndColumn(i, tilemap);
        SpriteAtlas.drawSprite(
          ~atlas=tilemap.atlas,
          ~ctx,
          ~atlasNumber=num,
          ~x=col * tilemap.tileSize,
          ~y=row * tilemap.tileSize,
        );
      },
      tilemap.tiles,
    );
};

Drawing the tilemap

Here we do the actual work. Setup the canvas, load the sprites, create a tilemap, then render it in the browser.

let doWindowOnload = () => {
  let canvas: Canvas.t = Canvas.getElementById(Document.t, "demo");
  let ctx: Context.t = Context.getContext2d(canvas);
  SpriteAtlas.make("./tiles16.png", 16)
  |> Js.Promise.then_(atlas => {
     let tiles = [ 
          2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
          2, 1, 1, 1, 1, 2, 5, 1, 1, 1, 1, 1, 1, 1, 1, 2, 
          2, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 
          2, 1, 1, 4, 1, 2, 2, 2, 2, 2, 0, 0, 0, 2, 1, 2,
          2, 1, 1, 1, 1, 2, 0, 0, 0, 0, 0, 0, 0, 2, 1, 2, 
          2, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 
          2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 
          0, 2, 1, 1, 1, 2, 1, 2, 2, 1, 1, 1, 1, 1, 1, 2, 
          0, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 3, 1, 1, 2, 
          0, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 2, 
          0, 2, 1, 3, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 
          0, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 
          0, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 2, 
          0, 0, 2, 1, 1, 3, 1, 1, 2, 1, 1, 1, 1, 5, 1, 2, 
          0, 0, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 2, 
          0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 
        ]; 

       let tilemap = 
         TileMap.make(~atlas, ~numRows=16, ~numCols=16, ~tileSize=16, ~tiles);

       TileMap.render(ctx, tilemap);

       Js.Promise.resolve();
     })
  |> Js.Promise.catch(err => {
       Js.log2("Failure!", err);
       Js.Promise.resolve();
     })
  |> ignore;
};

Window.addEventListener(Window.t, "load", doWindowOnload);

Exploring Geospatial data in Elixir with Phoenix, D3, and PostGIS

Thu, 28 Jan 2016 00:00:00 +0000

I noticed that Kaggle has started hosting some publicly available datasets. One of which is Ocean Ship Logbooks (1750-1850). I thought it would be interesting to visualize Captain Cook's travels on the HMS Endeavour using the Phoenix Web Framework, the D3 JavaScript library, and PostGIS.

Source code is available at this GitHub repo.

Captain Cook's Journeys on the Endeavor

Docker

For simplicity in this example we'll be using the mdillon/postgis Docker image. If you already have PostGIS installed or do not want to use Docker feel free to skip this section.

# Create a persistent data volume
$ sudo docker create -v /var/lib/postgresql/data \
    --name postgres-ocean-ships-data busybox

# Create the postgis container
$ sudo docker run --name postgres-ocean-ships -p 5432:5432 \
    -e POSTGRES_PASSWORD=postgres -d --volumes-from postgres-ocean-ships-data \
    mdillon/postgis:9.4

You can double check everything is working by connecting to Postgres. When prompted, the password will be "postgres", but you should consider using a much better one.

If you have psql installed locally you can do it like so:

$ psql -h localhost -p 5432 -U postgres

Otherwise you can also use the one in the container and connect with these commands:

$ sudo docker exec -i -t postgres-ocean-ships bash
root@10499abd059e:/# psql -h localhost -p 5432 -U postgres

Now that the containers are set up, in the future you can start and stop the PostGIS container like this:

$ sudo docker start postgres-ocean-ships
$ sudo docker stop postgres-ocean-ships

Phoenix

Create the Phoenix app:

$ mix phoenix.new ocean_ship_logbooks
$ cd ocean_ship_logbooks/
$ mix ecto.create

Add these Elixir CSV and Geo libraries to your mix.exs dependencies.

{:csv, "~> 1.2.3"}
{:geo, "~> 1.0"}

Fetch them

$ mix deps.get

We are going to be using be using the CSV library to parse the data, and the Geo library to work with PostGIS (via Ecto), so add to your config files (dev.exs, test.exs, and prod.secret.exs):

extensions: [{Geo.PostGIS.Extension, library: Geo}]

For example, the database section of tour config/dev.exs should now look something like:

# Configure your database
config :ocean_ship_logbooks, OceanShipLogbooks.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "postgres",
  password: "postgres",
  database: "ocean_ship_logbooks_dev",
  hostname: "localhost",
  pool_size: 10,
  extensions: [{Geo.PostGIS.Extension, library: Geo}]

Create database migrations

First we will want to enable the PostGIS extension in Postgres for our database. Lets generate a migration.

$ mix ecto.gen.migration enable_postgis
* creating priv/repo/migrations
* creating priv/repo/migrations/20160128183445_enable_postgis.exs

Edit 20160128183445_enable_postgis.exs so that the contents are this:

defmodule OceanShipLogbooks.Repo.Migrations.EnablePostgis do
  use Ecto.Migration

  def up do
    execute "CREATE EXTENSION IF NOT EXISTS postgis"
  end

  def down do
    execute "DROP EXTENSION IF EXISTS postgis"
  end

end

Now lets create ship data table.

Since the ship data will be accessed via a JSON API, you normally might generate a resource with the mix phoenix.gen.json task to create the model, migration, view, etc. code, but since we are using custom types from the Geo library we'll just create them manually.

$ mix ecto.gen.migration create_ship_data
* creating priv/repo/migrations
* creating priv/repo/migrations/20160128223253_create_ship_data.exs

We'll want to store the name of the ship, a timestamp, and a geometry object in the database, and we will be querying on the ship's name. Alter the file so that it has the following contents:

defmodule OceanShipLogbooks.Repo.Migrations.CreateShipData do
  use Ecto.Migration

  def change do
    create table(:ship_data) do
      add :ship,     :string
      add :utc,      :integer
      add :geom,     :geometry
    end
    create index(:ship_data, [:ship])
  end

end

Run the migrations:

$ mix ecto.migrate

ShipData Model

Create a file at web/models/ship_data.ex with these contents:

defmodule OceanShipLogbooks.ShipData do
  use OceanShipLogbooks.Web, :model
  schema "ship_data" do
    field :ship, :string
    field :utc,  :integer
    field :geom, Geo.Point
  end
end

This is a pretty straightforward model. We are using the Geo library so we can use a geometry column type that is specific to PostGIS.

ShipData View

Create a file web/views/ship_data_view.ex with these contents:

defmodule OceanShipLogbooks.ShipDataView do
  use OceanShipLogbooks.Web, :view

  def render("show_ships.json", %{ships: ships}) do
   %{ship_data: render_many(ships, OceanShipLogbooks.ShipDataView, "ship_data.json")}
  end

  def render("ship_data.json", %{ship_data: ship}) do
    {lat, lon} = ship.geom.coordinates

    %{ship: ship.ship,
      utc: ship.utc,
      lat: lat,
      lon: lon}
  end

end

Router

Look in web/router.ex for the api block. Uncomment it and change to look like this:

# Other scopes may use custom stacks.
scope "/api", OceanShipLogbooks do
  pipe_through :api
  resources "/ship_data", ShipDataController, only: [:index]
end

Ship Data Controller

Create a file web/controllers/ship_data_{controller.ex} with these contents:

defmodule OceanShipLogbooks.ShipDataController do
  use OceanShipLogbooks.Web, :controller
  alias OceanShipLogbooks.ShipData

  def index(conn, _params) do
    # For simplicity in this example we hard code against
    # a specific ship name.
    query = from s in ShipData,
    where: s.ship == "Endeavour",
    order_by: [asc: s.utc],
    select: s

    ships = Repo.all(query)
    render(conn, "show_ships.json", ships: ships)
  end

  # These next two functions probably do not belong in a controller,
  # since they are for manually importing data, but we put them in here
  # for this example.
  def import_from_csv() do
    File.stream!("ship-data.csv")
    |> CSV.decode(headers: true)
    |> Stream.filter(fn(x) -> x["Lat3"] != "NA" and x["Lon3"] != "NA" end)
    |> Stream.map(&build_ship/1)
    |> Enum.each(&OceanShipLogbooks.Repo.insert!/1)
  end

  defp build_ship(row) do
    {lat, _} = Float.parse(row["Lat3"])
    {lon, _} = Float.parse(row["Lon3"])
    {utc, _} = Integer.parse(row["UTC"])
    geom = %Geo.Point{ coordinates: {lat, lon}, srid: 4326}
    %OceanShipLogbooks.ShipData{ship: row["ShipName"], utc: utc, geom: geom}
  end

end

Download and clean the data.

Ensure you have downloaded the data the ship data from Kaggle. Extract the file CLIWOC15.csv into the root directory of your project (this file should be about 200 megabytes in size).

Unfortunately the Elixir CSV parser we are using is not RFC 4180 compliant and the ship data includes fields with escaped newlines. I did not investigate other Elixir or Erlang CSV parsers, so while not ideal, until this is fixed I removed the newlines with the following python script called transform_csv.py:

import csv

with open("CLIWOC15.csv", 'rU') as csvIN:
    with open('ship-data.csv', 'wb') as csvOUT:
        writer = csv.writer(csvOUT, delimiter=',', quoting=csv.QUOTE_ALL)
        for line in csv.reader(csvIN, delimiter=','):
            line = [x.replace('\n', '') for x in line]
            writer.writerow(line)

Run the script to remove extraneous newlines:

$ python transform-csv.py

Load the data into PostGIS

Run the app inside IEx (Interactive Elixir):

$ iex -S mix phoenix.server
iex(1)> OceanShipLogbooks.ShipDataController.import_from_csv
# Exit iex

Now verify the data is in PostGIS. Your session should look something like this:

$ psql -h localhost -p 5432 -U postgres ocean_ship_logbooks_dev

ocean_ship_logbooks_dev=# select count(*) from ship_data;
 count  
--------
 252917
(1 row)

ocean_ship_logbooks_dev=# select * from ship_data where ship = 'Endeavour' limit 4;
  id   |   ship    |    utc     |                        geom                        
-------+-----------+------------+----------------------------------------------------
 41085 | Endeavour | 1768110915 | 0101000020E610000024287E8CB97B35C03333333333D342C0
 41084 | Endeavour | 1768093011 | 0101000020E61000006666666666E62F40423EE8D9ACAA35C0
 41083 | Endeavour | 1768090513 | 0101000020E6100000545227A08988454000000000000024C0
 41082 | Endeavour | 1768090413 | 0101000020E6100000AE47E17A14CE4540D7A3703D0A5724C0

You can also query the data via the phoenix app:

$ iex -S mix phoenix.server

iex(1)> import Ecto.Query, only: [from: 2]
iex(2)> query = from s in OceanShipLogbooks.ShipData,
...(2)> where: s.ship == "Endeavour",
...(2)> limit: 4,
...(2)> select: s
iex(3)> ships = OceanShipLogbooks.Repo.all(query)
[%OceanShipLogbooks.ShipData{__meta__: #Ecto.Schema.Metadata<:loaded>,
  geom: %Geo.Point{coordinates: {-21.4833, -37.65}, srid: 4326}, id: 41085,
  ship: "Endeavour", utc: 1768110915},
 %OceanShipLogbooks.ShipData{__meta__: #Ecto.Schema.Metadata<:loaded>,
  geom: %Geo.Point{coordinates: {15.95, -21.6667}, srid: 4326}, id: 41084,
  ship: "Endeavour", utc: 1768093011},
 %OceanShipLogbooks.ShipData{__meta__: #Ecto.Schema.Metadata<:loaded>,
  geom: %Geo.Point{coordinates: {43.0667, -10.0}, srid: 4326}, id: 41083,
  ship: "Endeavour", utc: 1768090513},
 %OceanShipLogbooks.ShipData{__meta__: #Ecto.Schema.Metadata<:loaded>,
  geom: %Geo.Point{coordinates: {43.61, -10.17}, srid: 4326}, id: 41082,
  ship: "Endeavour", utc: 1768090413}]

Feel free to delete the CSV files to clear up a few hundred megs of disk space.

Download Countries GeoJSON

Get the GeoGSON data from this GitHub repo and put the geojson file at priv/static/js/world.geojson

This will let us draw a map of the world in D3.

HTML and Javascript code

Edit web/templates/page/index.html.eex to look like this:


  Captain Cook's travels

Now start the server

$ mix phoenix.server

Open a web browser to http://localhost:4000 to see an animation like at the top of this post and a page that should look something like this:

Getting started with OpenGL in Elixir

Sun, 03 Jan 2016 00:00:00 +0000

I was curious about using OpenGL with Elixir and couldn't find much very information. Other than a kludge for working with constants defined in Erlang .hrl files it turned out to be not too difficult.

The final result draws a triangle on the screen and looks like this:

wxWidgets and Erlang

Erlang/OTP ships with a port of wxWidgets, so creating a window and obtaining an OpenGL context is relatively easy.

To see some examples and the associated Erlang code start iex and run the following command:

iex(1)> :wx.demo

A window that looks something like this should appear.

Next we want to do something similar, but in Elixir instead.

Create an Elixir project

Run these commands in a shell

$ mix new elixir_opengl
$ cd elixir_opengl

# To make this work we are also going to need to create a couple Erlang source files
$ mkdir src

Erlang modules

If you look inside your Erlang/OTP distribution directory at the file lib/wx/include/wx.hrl you will see a number of defines that look like:

-define(wxID_ANY, -1).

I found this discussion about how to access them from Elixir - create an Erlang module with functions that return the value. You can then reference it from Elixir with a function call like this:

:wx_const.wx_id_any()

Now create an Erlang file named src/wx_const.erl with the following contents:

-module(wx_const).
-compile(export_all).

-include_lib("wx/include/wx.hrl").

wx_id_any() ->
  ?wxID_ANY.

wx_sunken_border() ->
  ?wxSUNKEN_BORDER.

wx_gl_rgba() ->
  ?WX_GL_RGBA.

wx_gl_doublebuffer() ->
 ?WX_GL_DOUBLEBUFFER.

wx_gl_min_red() ->
  ?WX_GL_MIN_RED.

wx_gl_min_green() ->
  ?WX_GL_MIN_GREEN.

wx_gl_min_blue() ->
  ?WX_GL_MIN_BLUE.

wx_gl_depth_size() ->
  ?WX_GL_DEPTH_SIZE.

Next create a file src/gl_const.erl with these contents:

-module(gl_const).
-compile(export_all).

-include_lib("wx/include/gl.hrl").

gl_smooth() ->
  ?GL_SMOOTH.

gl_depth_test() ->
  ?GL_DEPTH_TEST.

gl_lequal() ->
  ?GL_LEQUAL.

gl_perspective_correction_hint() ->
  ?GL_PERSPECTIVE_CORRECTION_HINT.

gl_nicest() ->
  ?GL_NICEST.

gl_color_buffer_bit() ->
  ?GL_COLOR_BUFFER_BIT.

gl_depth_buffer_bit() ->
  ?GL_DEPTH_BUFFER_BIT.

gl_triangles() ->
  ?GL_TRIANGLES.

gl_projection() ->
  ?GL_PROJECTION.

gl_modelview() ->
  ?GL_MODELVIEW.

We're now done with Erlang.

Elixir Code

Change lib/elixir_opengl.ex to look like this:

defmodule ElixirOpengl do
  @behaviour :wx_object
  use Bitwise

  @title 'Elixir OpenGL'
  @size {600, 600}

  #######
  # API #
  #######
  def start_link() do
    :wx_object.start_link(__MODULE__, [], [])
  end

  #################################
  # :wx_object behavior callbacks #
  #################################
  def init(config) do
    wx = :wx.new(config)
    frame = :wxFrame.new(wx, :wx_const.wx_id_any, @title, [{:size, @size}])
    :wxWindow.connect(frame, :close_window)
    :wxFrame.show(frame)

    opts = [{:size, @size}]
    gl_attrib = [{:attribList, [:wx_const.wx_gl_rgba,
                                :wx_const.wx_gl_doublebuffer,
                                :wx_const.wx_gl_min_red, 8,
                                :wx_const.wx_gl_min_green, 8,
                                :wx_const.wx_gl_min_blue, 8,
                                :wx_const.wx_gl_depth_size, 24, 0]}]
    canvas = :wxGLCanvas.new(frame, opts ++ gl_attrib)

    :wxGLCanvas.connect(canvas, :size)
    :wxWindow.reparent(canvas, frame)
    :wxGLCanvas.setCurrent(canvas)
    setup_gl(canvas)

    # Periodically send a message to trigger a redraw of the scene
    timer = :timer.send_interval(20, self(), :update)

    {frame, %{canvas: canvas, timer: timer}}
  end

  def code_change(_, _, state) do
    {:stop, :not_implemented, state}
  end

  def handle_cast(msg, state) do
    IO.puts "Cast:"
    IO.inspect msg
    {:noreply, state}
  end

  def handle_call(msg, _from, state) do
    IO.puts "Call:"
    IO.inspect msg
    {:reply, :ok, state}
  end

  def handle_info(:stop, state) do
    :timer.cancel(state.timer)
    :wxGLCanvas.destroy(state.canvas)
    {:stop, :normal, state}
  end

  def handle_info(:update, state) do
    :wx.batch(fn -> render(state) end)
    {:noreply, state}
  end

  # Example input:
  # {:wx, -2006, {:wx_ref, 35, :wxFrame, []}, [], {:wxClose, :close_window}}
  def handle_event({:wx, _, _, _, {:wxClose, :close_window}}, state) do
    {:stop, :normal, state}
  end

  def handle_event({:wx, _, _, _, {:wxSize, :size, {width, height}, _}}, state) do
    if width != 0 and height != 0 do
      resize_gl_scene(width, height)
    end
    {:noreply, state}
  end

  def terminate(_reason, state) do
    :wxGLCanvas.destroy(state.canvas)
    :timer.cancel(state.timer)
    :timer.sleep(300)
  end

  #####################
  # Private Functions #
  #####################
  defp setup_gl(win) do
    {w, h} = :wxWindow.getClientSize(win)
    resize_gl_scene(w, h)
    :gl.shadeModel(:gl_const.gl_smooth)
    :gl.clearColor(0.0, 0.0, 0.0, 0.0)
    :gl.clearDepth(1.0)
    :gl.enable(:gl_const.gl_depth_test)
    :gl.depthFunc(:gl_const.gl_lequal)
    :gl.hint(:gl_const.gl_perspective_correction_hint, :gl_const.gl_nicest)
    :ok
  end

  defp resize_gl_scene(width, height) do
    :gl.viewport(0, 0, width, height)
    :gl.matrixMode(:gl_const.gl_projection)
    :gl.loadIdentity()
    :glu.perspective(45.0, width / height, 0.1, 100.0)
    :gl.matrixMode(:gl_const.gl_modelview)
    :gl.loadIdentity()
    :ok
  end

  defp draw() do
    :gl.clear(Bitwise.bor(:gl_const.gl_color_buffer_bit, :gl_const.gl_depth_buffer_bit))
    :gl.loadIdentity()
    :gl.translatef(-1.5, 0.0, -6.0)
    :gl.'begin'(:gl_const.gl_triangles)
    :gl.vertex3f(0.0, 1.0, 0.0)
    :gl.vertex3f(-1.0, -1.0, 0.0)
    :gl.vertex3f(1.0, -1.0, 0.0)
    :gl.'end'()
    :ok
  end

  defp render(%{canvas: canvas} = _state) do
    draw()
    :wxGLCanvas.swapBuffers(canvas)
    :ok
  end
end

Now lets run the program, start iex like this:

$ iex -S mix

And run this command

iex(1)> ElixirOpengl.start_link

A window should appear that looks like this:

I suspect this can be improved on, but we've got an OpenGL context which is good enough for now.

Erlang clustering on the Raspberry Pi and BeagleBone with Elixir

Fri, 11 Dec 2015 00:00:00 +0000

I have been looking at Elixir/Erlang clustering with credit card sized computers and using the Phoenix web framework to communicate from the cluster to an Android tablet in realtime.

I am pleasantly surprised at how easy it has been and how little code I had to write for this demo.

In the above video I am running an Erlang cluster consisting of a desktop x86 based PC on Ubuntu, a Raspberry Pi, a BeagleBone Black, and an Android tablet. Each of the nodes are running Erlang/OTP 18.1 and Elixir 1.1.1.

A process written in Elixir on the PC periodically communicates with the Raspberry Pi and BeagleBone to turn on and off LEDs connected via a GPIO pin to a breadboard, and concurrently sends a message to the tablet over a WebSocket in realtime.

Android code

I simply took this Android Phoenix Demo on GitHub, followed the instructions to update the host value, and installed in on a Nexus 7.

Raspberry Pi and BeagleBone Code

On both your Pi and BeagleBone create a file called gpio-led.ex with the following contents:

defmodule GPIO.LED do
  use GenServer

  #####
  # External API
  def start_link(pin) do
    :os.cmd(to_char_list "echo #{pin} > /sys/class/gpio/export")
    :os.cmd(to_char_list "echo out > /sys/class/gpio/gpio#{pin}/direction")
    :os.cmd(to_char_list "echo 0 > /sys/class/gpio/gpio#{pin}/value")
    GenServer.start_link(__MODULE__, pin, name: __MODULE__)
  end

  def led_on! do
    GenServer.cast __MODULE__, :led_on
  end

  def led_off! do
    GenServer.cast __MODULE__, :led_off
  end

  #####
  # GenServer implementation
  def handle_cast(:led_on, pin) do
    :os.cmd(to_char_list "echo 1 > /sys/class/gpio/gpio#{pin}/value")
    {:noreply,  pin}
  end

  def handle_cast(:led_off, pin) do
    :os.cmd(to_char_list "echo 0 > /sys/class/gpio/gpio#{pin}/value")
    {:noreply,  pin}
  end
end

This is a pretty straightforward GenServer. For simplicity I am not using any supervisors in this example, i'll leave that as an exercise for the reader.

If you are curious about what is happening with the GPIO and want to learn more I have written up some information about getting started with Erlang/Elixir and GPIO on the BeagleBone and Raspberry Pi.

Web server code

I started with the Phoenix chat example on GitHub. Clone the repo to your desktop or laptop.

Next create a file called phoenix_chat_example/lib/chat/led_{controller.ex} with the following contents:

defmodule Chat.LedController do

  def blink_leds() do
    # Ensure we are connected
    true = Node.connect :"beagle@beaglebone.local"
    true = Node.connect :"pi@raspberrypi.local"

    # Ensure the GPIO.LED servers have started on the Raspberry Pi and BeagleBone
    :rpc.call(:"beagle@beaglebone.local", GPIO.LED, :start_link, [23])
    :rpc.call(:"pi@raspberrypi.local", GPIO.LED, :start_link, [4])

    loop()
  end

  defp loop() do
    Chat.Endpoint.broadcast! "rooms:lobby", "new:msg", %{user: "server",
                                                         body: "BeagleBone ON"}
    :rpc.call(:"beagle@beaglebone.local", GPIO.LED, :led_on!, [])
    :timer.sleep(1000)
    :rpc.call(:"beagle@beaglebone.local", GPIO.LED, :led_off!, [])

    Chat.Endpoint.broadcast! "rooms:lobby", "new:msg", %{user: "server",
                                                         body: "Pi ON"}
    :rpc.call(:"pi@raspberrypi.local", GPIO.LED, :led_on!, [])
    :timer.sleep(1000)
    :rpc.call(:"pi@raspberrypi.local", GPIO.LED, :led_off!, [])

    loop()
  end
end

This is also straightforward.

Send a message to a Phoenix Channel (which in turn gets forwarded to the Android app via websocket).
Periodically make rpc calls to the nodes on the Raspberry Pi and Beaglebone to turn on and off the LEDs
Sleep for a second
Repeat

Once started the code will recursively loop forever.

Run the code

ssh to the Raspberry Pi as the root user

# Start an Erlang node
root@raspberrypi:~# iex --name pi@raspberrypi.local --cookie peanut-butter

# Compile
iex(pi@raspberrypi.local)1> c("gpio-led.ex")

ssh to the BeagleBone as the root user

root@beaglebone:~# iex --name beagle@beaglebone.local --cookie peanut-butter
iex(beagle@beaglebone.local)1> c("gpio-led.ex")

On your desktop or laptop run the code as an Elixir Task :

# Start the Phoenix app
$ iex --name desktop@desktop.local --cookie peanut-butter -S mix phoenix.server

# Blink the LEDs
iex(1)> {:ok, pid} = Task.start(fn -> Chat.LedController.blink_leds end)

The LEDs should be turning on and off and you should be seeing messages from Elixir in the Android app.

Elixir on the Raspberry Pi - Blinking an LED

Thu, 10 Dec 2015 00:00:00 +0000

In this post we will see how to run the "Hello, World" of embedded devices, blinking an LED on a breadboard. We will be using the Elixir programming language and a Raspberry Pi.

I have also written up similar instructions for the BeagleBone Black.

The final result will look like this:

Elixir programming language

Elixir describes itself:

Elixir is a dynamic, functional language designed for building scalable and
maintainable applications.

Elixir leverages the Erlang VM, known for running low-latency, distributed and
fault-tolerant systems, while also being successfully used in web development and
the embedded software domain.

Some of Elixir's selling points are:

Runs on the rock solid Erlang Virtual Machine.
Offers trivial interop with Erlang code, libraries, and frameworks including the OTP.
Code looks a fair bit like Ruby which many may find easier to grok than Erlang's Prolog inspired syntax.

So far it has been an absolute joy to use.

Raspberry Pi

The Raspberry Pi is a series of small (credit card sized) and relatively low cost computers capable of running Linux with a number of general-purpose input/output (GPIO) pins which can interface with the outside world.

_{Image by Lucasbosch (Own work) [CC BY-SA 3.0], via Wikimedia Commons}

Now lets install the software dependencies.

Install Erlang and Elixir

Elixir currently requires Erlang 17.0 or later, the debian packages in the default repositories are too old so we will need an alternative installation method.

One possibility would be to use the nerves project. I recently watched the ElixirConf 2015 - Embedded Elixir in Action by Garth Hitchens presentation and am excited to see where it goes, but for this example it would probably be overkill.

On a desktop PC another option is to use one of the Erlang Solutions repos, but as far as I can tell the debian packages are only built for x86 architectures and the ones for Raspbian have not been updated since Erlang 15.

So it looks like we'll be installing from source.

# ssh in and run these commands on your Raspberry Pi
# It should take you about an hour from start to finish

# Download, compile, and install Erlang
$ apt-get install wget libssl-dev ncurses-dev m4 unixodbc-dev erlang-dev
$ wget http://www.erlang.org/download/otp_src_18.1.tar.gz
$ tar -xzvf otp_src_18.1.tar.gz
$ cd otp_src_18.1/
$ export ERL_TOP=`pwd`
$ ./configure
$ make
$ make install

# Download a precompiled elixir release from
# https://github.com/elixir-lang/elixir/releases/
$ apt-get install unzip
$ wget https://github.com/elixir-lang/elixir/releases/download/v1.1.1/Precompiled.zip
$ unzip Precompiled.zip -d elixir

# Add elixir to your path
# You may want to add this to your .bashrc so you do not have to every time you
# log on
$ export PATH="/home/pi/elixir/bin:$PATH"

# Ensure Elixir is working
$ iex
Erlang/OTP 18 [erts-7.1] [source] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.1.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 1 + 2
3
iex(2)>
# Press Ctrl+C twice to exit iex

We're now ready to explore GPIO (general purpose I/O) on the Pi.

Wire up the LED

Note: The GPIO pins can only handle 3.3 volts, so be very careful that you do not accidentally connect a jumper to one of the 5 volt source pins. If you are unsure of what you are doing I would highly recommend reading all of the Raspberry Pi documentation to make sure you do not damage your board.

Using a jumper wire connect GND (ground) on the Pi to the negative rail on the breadboard.
Place an LED in the breadboard.
Using a jumper wire connect the cathode (shorter) wire of the LED to the negative rail.
Connect one end of a 100 ohm resistor to the anode (longer) wire of the LED.
Using another jumper wire connect the other end of the resistor to GPIO 4.

Once wired up we can proceed.

Turn an LED on and off on the command line

In this example we'll be using the built in sysfs to control the GPIO pins. Wikipedia describes it:

sysfs is a virtual file system provided by the Linux kernel that exports information
about various kernel subsystems, hardware devices, and associated device drivers
from the kernel's device model to user space through virtual files. In addition to
providing information about various devices and kernel subsystems, exported virtual
files are also used for their configuring.

This allows us to treat the pins like a file, and while not the most efficient way to work with the GPIO pins, it is very convenient for some use cases.

When you follow along with the example below on a Pi it should make sense.

# To make it easier in this example we run commands as root
pi@raspberrypi ~ $ sudo -i

# Add elixir to your root user's path
# You may want to add this to your .bashrc so you do not have to every time you
# log on
root@raspberrypi:~# export PATH="/home/pi/elixir/bin:$PATH"

# One gpio pin already exists
root@raspberrypi:~# ls /sys/class/gpio
export  gpiochip0  unexport

# Export a new one
root@raspberrypi:~# echo 4 > /sys/class/gpio/export

# Notice that gpio4 has now appeared
root@raspberrypi:~# ls /sys/class/gpio
export  gpio4  gpiochip0  unexport

# What we can do with gpio4
root@raspberrypi:~# ls /sys/class/gpio/gpio4
active_low  device  direction  edge  subsystem  uevent  value

# Set the direction to out
root@raspberrypi:~# echo out > /sys/class/gpio/gpio4/direction

# Turn on the LED
root@raspberrypi:~# echo 1 > /sys/class/gpio/gpio4/value

# Turn off the LED
root@raspberrypi:~# echo 0 > /sys/class/gpio/gpio4/value

# Now use Elixir to turn the LED on and then off
root@raspberrypi:~# iex
iex(1)> :os.cmd('echo 1 > /sys/class/gpio/gpio4/value')
[]
iex(2)> :os.cmd('echo 0 > /sys/class/gpio/gpio4/value')
[]

# Press Ctrl+C twice to exit iex

# Clean up
root@raspberrypi:~# echo 4 > /sys/class/gpio/unexport

Now lets make a module.

Elixir Code

On your Raspberry Pi create a file called blink-led.ex with the following contents:

defmodule BlinkLED do
  @moduledoc """
  Blink an LED on a Raspberry Pi

  ## Examples

  iex> c("blink-led.ex")
  [BlinkLED]
  iex> {:ok, pid} = BlinkLED.start_link()
  {:ok, #PID<0.102.0>}

  """

  @doc """
  Setup and start the process
  """
  def start_link() do
    :os.cmd('echo 4 > /sys/class/gpio/export')
    :os.cmd('echo out > /sys/class/gpio/gpio4/direction')
    {:ok, spawn_link(fn -> loop() end)}
  end

  defp loop() do
    :os.cmd('echo 1 > /sys/class/gpio/gpio4/value')
    :timer.sleep(1000)
    :os.cmd('echo 0 > /sys/class/gpio/gpio4/value')
    :timer.sleep(1000)
    loop()
  end
end

To run the code you can start iex and do the following:

iex> c("blink-led.ex")
iex> {:ok, pid} = BlinkLED.start_link()

The LED on the breadboard should now turn on for one second, turn off for a second, and repeat indefinitely.

Elixir on the BeagleBone Black - Blinking an LED

Wed, 09 Dec 2015 00:00:00 +0000

In this post we will see how to run the "Hello, World" of embedded devices, blinking an LED on a breadboard. We will be using the Elixir programming language and a BeagleBone Black.

I have also written up similar instructions for the Raspberry Pi.

The final result will look like this:

Elixir programming language

Elixir describes itself:

Elixir is a dynamic, functional language designed for building scalable and
maintainable applications.

Elixir leverages the Erlang VM, known for running low-latency, distributed and
fault-tolerant systems, while also being successfully used in web development and
the embedded software domain.

Some of Elixir's selling points are:

Runs on the rock solid Erlang Virtual Machine.
Offers trivial interop with Erlang code, libraries, and frameworks including the OTP.
Code looks a fair bit like Ruby which many may find easier to grok than Erlang's Prolog inspired syntax.

So far it has been an absolute joy to use.

BeagleBone Black

The BeagleBone Black is a small and relatively low cost 1Ghz ARM board with 512Mb of RAM capable of running Linux. It is similar to a Raspberry Pi (and most of the information here will be applicable to a Pi as well).

These instructions assume that your BeagleBone is using Debian 2015-03-01 and running commands as the root user. You can obtain this version of the operating system here.

If unsure what version is running you can determine it by running this command on the board:

$ cat /etc/dogtag
BeagleBoard.org Debian Image 2015-03-01

Now lets install the software dependencies.

Install Erlang and Elixir

Elixir currently requires Erlang 17.0 or later, the debian packages in the default repositories are too old so we will need an alternative installation method.

On a desktop PC another option is to use one of the Erlang Solutions repos, but as far as I can tell they only provide recent versions of Erlang built for x86 architectures and the BeagleBone uses an ARM chip.

So it looks like we'll be installing from source.

# ssh in and run these commands on your BeagleBone Black
# It should take you about an hour from start to finish

# Download, compile, and install Erlang
$ apt-get install wget libssl-dev ncurses-dev m4 unixodbc-dev erlang-dev
$ wget http://www.erlang.org/download/otp_src_18.1.tar.gz
$ tar -xzvf otp_src_18.1.tar.gz
$ cd otp_src_18.1/
$ export ERL_TOP=`pwd`
$ ./configure
$ make
$ make install

# Download a precompiled elixir release from
# https://github.com/elixir-lang/elixir/releases/
$ apt-get install unzip
$ wget https://github.com/elixir-lang/elixir/releases/download/v1.1.1/Precompiled.zip
$ unzip Precompiled.zip -d elixir

# Add elixir to your path
# You may want to add this to your .bashrc so you do not have to every time you
# log on
$ export PATH="$HOME/elixir/bin:$PATH"

# Ensure Elixir is working
$ iex
Erlang/OTP 18 [erts-7.1] [source] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.1.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 1 + 2
3
iex(2)>
# Press Ctrl+C twice to exit iex

We're now ready to explore GPIO (general purpose I/O) on the BeagleBone.

Wire up the LED

Note: The GPIO pins can only handle 3.3 volts, so be very careful that you do not accidentally connect a jumper to one of the 5 volt source pins. If you are unsure of what you are doing I would highly recommend reading the BeagleBone System Reference Manual to make sure you do not damage your board.

Using a jumper wire connect Pin 2 on Header P9 (ground) on the BeagleBone to the negative rail on the breadboard.
Place an LED in the breadboard.
Using a jumper wire connect the cathode (shorter) wire of the LED to the negative rail.
Connect one end of a 100 ohm resistor to the anode (longer) wire of the LED.
Using another jumper wire connect the other end of the resistor to Pin 13 on Header P8

Once wired up we can proceed.

Turn an LED on and off on the command line

In this example we'll be using the built in sysfs to control the GPIO pins. Wikipedia describes it:

sysfs is a virtual file system provided by the Linux kernel that exports information
about various kernel subsystems, hardware devices, and associated device drivers
from the kernel's device model to user space through virtual files. In addition to
providing information about various devices and kernel subsystems, exported virtual
files are also used for their configuring.

This allows us to treat the pins like a file, and while not the most efficient way to work with the GPIO pins, it is very convenient for some use cases.

When you follow along with the example below on a BeagleBone it should make sense.

# A few gpio pins already exist
$ ls /sys/class/gpio
export gpiochip0  gpiochip32  gpiochip64  gpiochip96  unexport

# Export a new one
$ echo 23 > /sys/class/gpio/export

# Notice that gpio23 has now appeared
$ ls /sys/class/gpio
export gpio23  gpiochip0  gpiochip32  gpiochip64  gpiochip96  unexport

# What we can do with gpio23
$ ls /sys/class/gpio/gpio23
active_low  direction  edge  power  subsystem  uevent  value

# Set the direction to out
$ echo out > /sys/class/gpio/gpio23/direction

# Turn on the LED
$ echo 1 > /sys/class/gpio/gpio23/value

# Turn off the LED
$ echo 1 > /sys/class/gpio/gpio23/value

# Now use Elixir to turn the LED on and then off
$ iex
iex(1)> :os.cmd('echo 1 > /sys/class/gpio/gpio23/value')
[]
iex(2)> :os.cmd('echo 0 > /sys/class/gpio/gpio23/value')
[]

Now lets make a module.

Elixir Code

On your BeagleBone create a file called blink-led.ex with the following contents:

defmodule BlinkLED do
  @moduledoc """
  Blink an LED on a BeagleBone Black

  ## Examples

  iex> c("blink-led.ex")
  [BlinkLED]
  iex> {:ok, pid} = BlinkLED.start_link()
  {:ok, #PID<0.102.0>}

  """

  @doc """
  Setup and start the process
  """
  def start_link() do
    :os.cmd('echo 23 > /sys/class/gpio/export')
    :os.cmd('echo out > /sys/class/gpio/gpio23/direction')
    {:ok, spawn_link(fn -> loop() end)}
  end

  defp loop() do
    :os.cmd('echo 1 > /sys/class/gpio/gpio23/value')
    :timer.sleep(1000)
    :os.cmd('echo 0 > /sys/class/gpio/gpio23/value')
    :timer.sleep(1000)
    loop()
  end
end

To run the code you can start iex and do the following:

iex> c("blink-led.ex")
iex> {:ok, pid} = BlinkLED.start_link()

The LED on the breadboard should now turn on for one second, turn off for a second, and repeat indefinitely.

Creating an Apache Commons Daemon using Component with Clojure

Thu, 29 Oct 2015 00:00:00 +0000

Introduction

One way to write a long running server application that runs on the JVM with proper start/stop semantics is to the the Apache Commons Daemon. In this post we'll see how we can do so in Clojure and also use the Component framework to structure our application.

The full source for this post is available on Github.

Component

Component describes itself as:

A tiny Clojure framework for managing the lifecycle and dependencies
of software components which have runtime state.

This is primarily a design pattern with a few helper functions. It can be seen as a
style of dependency injection using immutable data structures.

There are two concepts to be aware of:

Components

A collection of functions or procedures which share some runtime state.

Systems

Components are composed into systems. A system is a component which knows how to
start and stop other components. It is also responsible for injecting dependencies
into the components which need them.

I won't go into very much detail about Component in this post, to learn more the documentation is excellent.

The Application

We will be building an app with two components and one system.

The first component will store application metrics and the second component will repeatedly print "tick" to stdout and increment a tick counter in the metrics component once a second.

The system will bring them together and handle the metrics component being a dependency of the tick one.

Leiningen Project

Our project.clj looks like this:

(defproject apache-commons-daemon "0.1.0-SNAPSHOT"
  :description "Example of running a daemon"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.7.0"]
                 [commons-daemon/commons-daemon "1.0.15"]
                 [com.stuartsierra/component "0.3.0"]]
  :main apache-commons-daemon.core
  :aot :all)

Code

Next create a file called src/apache_commons_daemon/core.clj

Insert the following at the top of the file:

(ns apache-commons-daemon.core
  (:require [com.stuartsierra.component :as component])
  (:import [org.apache.commons.daemon Daemon DaemonContext])
  (:gen-class
   :implements [org.apache.commons.daemon.Daemon]))

Metrics Component

For simplicity the first component simply stores the number of times the other component has ticked, however we could easily store additional metrics.

We also provide a constructor and a function to increment the tick count.

;; Application Metrics Component
(defrecord MetricsComponent [num-ticks]
  ;; Implement the Lifecycle protocol
  component/Lifecycle
  (start [component]
    (println ";; Starting MetricsComponent")
    (reset! (:num-ticks component) 0)
    component)
  (stop [component]
    (println ";; Stopping MetricsComponent")
    (println (str ";; Metric num ticks: " @(:num-ticks component)))
    component))

(defn make-metrics-component
  "Constructor for a metrics component"
  []
  (map->MetricsComponent {:num-ticks (atom 0)}))

(defn inc-num-ticks-metric [metrics-component]
  (swap! (get metrics-component :num-ticks) inc))

Tick Component

At startup the tick component creates a future which repeatedly outputs "tick" to stdout for as long as the component's state is set to :running.

;; Tick Component
(defn do-ticks
  "Print to stdout and increment the ticks metric once a second when the component
  is running"
  [state metrics-component]
  (while (= :running @state)
    (println "tick")
    (inc-num-ticks-metric metrics-component)
    (Thread/sleep 1000)))

(defrecord TickComponent [state metrics-component]
  ;; Implement the Lifecycle protocol
  component/Lifecycle
  (start [component]
    (println ";; Starting TickComponent")
    (reset! (:state component) :running)
    ;; Do some work in another thread
    (future (do-ticks (:state component) metrics-component))
    component)
  (stop [component]
    (println ";; Stopping TickComponent")
    (reset! (:state component) :stopped)
    component))

(defn make-tick-component
  "Constructor for a tick component"
  []
  (map->TickComponent {:state (atom :stopped)}))

Application System

Here we have a system which creates the two components. It also indicates that the metrics component is a dependency of the tick component, so it will ensure the metrics component is started first and provide an instance of it to the tick component.

;; Application System
(defn make-app-system []
  (component/system-map
   :metrics-component (make-metrics-component)
   :tick-component (component/using
                    (make-tick-component)
                    [:metrics-component])))

Now we create the system and provide functions to start and stop it. The start and stop methods make it easy to develop and test in a REPL (without the need for jsvc in that environment).

(def app-system (make-app-system))

;; Separate start/stop functions for easier development in a REPL
(defn start []
  (alter-var-root #'app-system component/start))

(defn stop []
  (alter-var-root #'app-system component/stop))

Commons Daemon

Finally we provide an implementatation of the Daemon interface. This will allow us to start and stop the application with jsvc.

;; Commons Daemon implementatation
(defn -init [this ^DaemonContext context])

(defn -start [this]
  (start))

(defn -stop [this]
  (stop))

(defn -destroy [this])

Running the Application

Build with Leiningen:

$ lein uberjar

To interact with the daemon you will need jsvc as well. On Ubuntu 14.04 it is as easy as:

$ sudo apt-get install jsvc

Start/stop like this (change your java home to what is appropriate for you):

$ sudo /usr/bin/jsvc -java-home /usr/lib/jvm/java-8-oracle/jre/ \
  -cp "$(pwd)/target/apache-commons-daemon-0.1.0-SNAPSHOT-standalone.jar" \
  -outfile "$(pwd)/out.txt" \
  apache_commons_daemon.core

# Wait a few seconds...

$ sudo /usr/bin/jsvc -java-home /usr/lib/jvm/java-8-oracle/jre/ \
  -cp "$(pwd)/target/apache-commons-daemon-0.1.0-SNAPSHOT-standalone.jar" \
  -stop \
  apache_commons_daemon.core

$ sudo cat out.txt

You should see output that looks something like this:

;; Starting MetricsComponent
;; Starting TickComponent
tick
tick
tick
tick
tick
;; Stopping TickComponent
;; Stopping MetricsComponent
;; Metric num ticks: 5

Note that jsvc can be a bit finicky and you may need to make some changes depending on your environment. If you do not see anything in the file, try adding the -debug flag to the jsvc calls which will provide useful information.

Next Steps

You will likely want to do is create a script to start the application. There is much more to jsvc than we covered here, Sheldon Neilson has written an excellent article for Debian based systems which I encourage you to read.

Exploring Stack Exchange data with Clojure using Apache Spark and Flambo

Tue, 07 Jul 2015 00:00:00 +0000

Introduction

Apache Spark is a is a fast and general engine for large-scale data processing (as in terabytes or larger data sets), and Flambo is a Clojure DSL for working with Spark. Stack Exchange is a network of question and answer websites with a variety of topics (the most popular one being Stack Overflow). They periodically provide a creative commons licensed database dump. We'll be using data from the Stack Exchange Gaming Site as a toy dataset to work with them.

First we will use Spark to convert the Stack Exchange files provided in XML into Apache Parquet format and then later we will use it to run some queries to find out things like which users have the highest reputation, as well as which ones like to play Dwarf Fortress.

This post assumes you are using leiningen, some basic familiarity with either the Java or Scala Spark API, Spark 1.3.1 is installed in ~/bin/spark and that the March 2015 Gaming Stack Exchange Data Dump has been downloaded and extracted to ~/data/gaming-stackexchange The full code for this post is also available on Github.

Leiningen Project

Our project.clj looks like this:

(defproject flambo-gaming-stack-exchange "0.1.0-SNAPSHOT"
  :description "Example of using Spark and Flambo"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :profiles {:uberjar {:aot :all}
             :provided {:dependencies
                        [[org.apache.spark/spark-core_2.10 "1.3.1"]]}}
  :dependencies [[org.clojure/clojure "1.7.0"]
                 [yieldbot/flambo "0.6.0"]
                 [commons-codec/commons-codec "1.10"]
                 [org.clojure/data.xml "0.0.8"]
                 [org.apache.spark/spark-sql_2.10 "1.3.1"]])

Step 1: ETL Users

The Stack Exchange data is provided in XML format. While it would be possible to leave it as is and load the XML into one of Spark's Resilient Distributed Dataset (RDD) when we query (this might be desirable if you are building a data lake), here we will be building a data mart with a subset of the data, stored in Parquet format, and queried using Spark DataFrames.

We want to take a line from the Stack Exchange Users.xml file that looks something like this:

And convert it into a DataFrame represented like this:

| id | name         | reputation|
|----+--------------+-----------|
| 2  | Geoff Dalgas | 101       |

We'll use the following code:

(ns flambo-gaming-stack-exchange.etl-users
  (:require [flambo.conf :as conf]
            [flambo.api :as f]
            [flambo.sql :as sql]
            [clojure.data.xml :as xml])
  (:import [org.apache.spark.sql RowFactory]
           [org.apache.spark.sql.types StructType StructField Metadata DataTypes])
  (:gen-class))

(defn build-sql-context
  "Returns a Spark SQLContext"
  [app-name]
  (let [c (-> (conf/spark-conf)
              (conf/master "local[*]")
              (conf/app-name app-name))
        sc (f/spark-context c)]
    (sql/sql-context sc)))

(defn xml->row
  "Parse a row of user xml and return it as a Spark Row"
  [user-xml]
  (let [user (xml/parse-str user-xml)
        {{:keys [Id DisplayName Reputation]} :attrs} user]
    [(RowFactory/create (into-array Object [(Integer/parseInt Id) DisplayName  (Integer/parseInt Reputation)]))]))

"Spark function that reads in a line of XML and potentially returns a Row"
(f/defsparkfn parse-user
  [user-xml]
  (if (.startsWith user-xml  "  row user-xml)
    []))

(def user-schema
  (StructType.
   (into-array StructField [(StructField. "id" (DataTypes/IntegerType) true (Metadata/empty))
                            (StructField. "name" (DataTypes/StringType) true (Metadata/empty))
                            (StructField. "reputation" (DataTypes/IntegerType) true (Metadata/empty))])))

(defn -main [& args]
  (let [home (java.lang.System/getenv "HOME")
        sql-ctx (build-sql-context "ETL Users")
        sc (sql/spark-context sql-ctx)
        xml-users (f/text-file sc (str home "/data/gaming-stackexchange/Users.xml"))
        users (f/flat-map xml-users parse-user)
        users-df (.createDataFrame sql-ctx users user-schema)]
    (.saveAsParquetFile users-df (str home "/data/gaming-stack-exchange-warehouse/users.parquet"))))

Now build and run. For simplicity in these examples we will run everything in local mode.

$ lein uberjar
$ ~/bin/spark/bin/spark-submit --class flambo_gaming_stack_exchange.etl_users \
target/flambo-gaming-stack-exchange-0.1.0-SNAPSHOT-standalone.jar

You should now have a directory at ~/data/gaming-stack-exchange-warehouse/users.parquet containing the user data in Parquet format.

Step 2: ETL Posts

Next we want to load posts. This is almost identical to loading users. I'll omit most of the code here, but it is available on GitHub. However I will include the schema here to make following along with the queries easier.

(def post-schema
  (StructType.
   (into-array StructField
     [(StructField. "ownerId" (DataTypes/IntegerType) true (Metadata/empty))
      (StructField. "postType" (DataTypes/IntegerType) true (Metadata/empty))
      (StructField. "tags" (DataTypes/StringType) true (Metadata/empty))])))

Now ETL the posts into Parquet format.

$ ~/bin/spark/bin/spark-submit --class flambo_gaming_stack_exchange.etl_posts \
target/flambo-gaming-stack-exchange-0.1.0-SNAPSHOT-standalone.jar

Like with the users step, you should now have a directory at ~/data/gaming-stack-exchange-warehouse/posts.parquet containing the post data in Parquet format.

Step 3: Query the data

Now lets run some queries. The code here will be broken up and be pseudo-Clojure for demonstration purposes. The code this is derived from is available here at GitHub. And can be run like this:

$ ~/bin/spark/bin/spark-submit --class flambo_gaming_stack_exchange.core \
target/flambo-gaming-stack-exchange-0.1.0-SNAPSHOT-standalone.jar

It's worth noting that the gaming site data is not particularly large, in all likelihood you'd be better off loading the files into RAM with your programming language of choice or into a relational database and querying them that way. However if the data was significantly larger (terabytes or more) you'd be able to use the exact same code and horizontally scale your data processing over a cluster of machines.

Query 1 - Top users

Lets find the top 10 users by reputation who have at least a 30,000 reputation score.

(:require [flambo.conf :as conf]
          [flambo.api :as f]
          [flambo.sql :as sql])
(:import [org.apache.spark.sql Column]
         [org.apache.commons.codec.binary Hex])

(defn build-sql-context
  "Returns a Spark SQLContext"
  [app-name]
  (let [c (-> (conf/spark-conf)
              (conf/master "local[*]")
              (conf/app-name app-name))
        sc (f/spark-context c)]
    (sql/sql-context sc)))

(let [home (java.lang.System/getenv "HOME")
      sql-ctx (build-sql-context "Stack Exchange Queries")

      ;; Read in the users Parquet file
      users (sql/parquet-file sql-ctx (string-array (str home "/data/gaming-stack-exchange-warehouse/users.parquet")))

      ;; This is one way to query a DataFrame.
      query (-> users
                (.select (column-array (.col users "name") (.col users "reputation")))
                (.filter "reputation > 30000")
                (.orderBy (column-array (-> users
                                            (.col "reputation")
                                            (.desc))))
                (.limit 10))

      ;; This is another way to run the same query, but to do it this way
      ;; we must first register any tables we will be using.
      _ (sql/register-temp-table users "users")
      sql-query (sql/sql sql-ctx "SELECT name, reputation FROM users WHERE reputation > 30000 ORDER BY reputation DESC LIMIT 10")]
    (.show sql-query))

Running either of the the above queries will output:

name             reputation
Raven Dreamer    123648
agent86          89947
z  '             54792
LessPop_MoreFizz 51582
kalina           41496
Oak              40572
tzenes           40458
StrixVaria       38108
badp             37688
fredley          35584

Query 2 - Top users, names obfuscated

Next lets say we've been given the requirement that we need to obfuscate user names prior to displaying them. Here we will use a SHA-1 hash function and the Apache Commons Hex encoder.

This is obviously not a good way to obfuscate the name and not how you would want to do it in a production environment, but for the purposes of demonstrating calling arbitrary functions in Spark it is "good enough" for this example.

(defn hash-string
  "Returns a hexidecimal encoded SHA-1 hash of a string"
  [data]
  (-> (java.security.MessageDigest/getInstance "SHA-1")
      (.digest (.getBytes data))
      (Hex/encodeHexString)))

;; "Hash name in a row with the schema [name reputation], returning a
;;  vector with the name hashed."
(f/defsparkfn hash-name
  [row]
  (let [[name reputation] row
        hashed-name (hash-string name)]
    [hashed-name reputation]))

;; We can also turn a DataFrame to an RDD and use Flambo functions.
;; Here we hash the users name prior to displaying it.
rdd-query (-> sql-query
              (.toJavaRDD)
              (f/map sql/row->vec)
              (f/map hash-name)
              (f/foreach (f/fn [x] (println x))))

A few things to note:

sql-query is the DataFrame from the previous query example.
We need to turn the DataFrame into a RDD.
At this time not all of the Spark SQL functions are wrapped in Clojure, so we have to call the Java method .toJavaRDD directly on the DataFrame object rather than a function provided by Flambo.
We need to use Flambo's defsparkfn macro to to define the function we will be using.

Running the above code will output:

[74cfefcff37f81e278398e4de6ce6ada68e7c80e 123648]
[e6eaf112ecf8020e9e1d389ca1e432488bfc7476 89947]
[a8d6f6c81225d88c1797dfc0592935a879d3b626 54792]
[aaa0b19e5e4db300832368b88871e39b695693fe 51582]
[6109e96245bc6d375d942e1cd0d88da8c8172f4a 41496]
[7badddf11e798303d6321ad096d5b2b447f97293 40572]
[7572126b4fd86ccd3610c5280a116c51f186781f 40458]
[8af617f985b18b676f6d809d9b1a9615b72a187b 38108]
[e122e7f4d0fc01a52a909621b43e67ddab506889 37688]
[9f9b1f7ec647cec6f9a4477face2216a81fee0dc 35584]

Query 3 - People who like to play Dwarf Fortress

Finally lets run one more query to find out which users have created the most questions about the game Dwarf Fortress. In order to answer this we'll need to join users against posts and can do so like this:

;; Top 10 users by number of questions asked about the
;; game Dwarf Fortress.
(let [df-query (sql/sql sql-ctx
                        "SELECT u.name, count(1) as cnt
                        FROM users u, posts p
                        WHERE p.tags LIKE '%dwarf-fortress%'
                        AND u.id = p.ownerId
                        GROUP BY u.name
                        ORDER BY cnt DESC
                        LIMIT 10")]
  (.show df-query))

Running the query gives us these users and how many questions they have asked:

name           cnt
antony.trupe   37
aslum          32
Anna           31
C. Ross        29
Paralytic      15
David Grinberg 15
user5781       14
Mechko         13
andronikus     10
Menno Gouw     9

Conclusion

While this post looked mostly at Spark SQL and DataFrames, support for them in Flambo is currently a bit rudimentary. It still provides access to the underlying Java objects, and the examples above demonstrated this is not a problem. I suspect more of it may get wrapped in Clojure land in future releases.

Flambo seems to be currently geared more towards working with Resilient Distributed Datasets. There are a number of RDD operations we didn't touch on here. I highly recommend exploring the documentation to see more of what is available.

Adventures in Clojure with core.async - Part 2 - Timeouts and Working with Multiple Channels via Parking and alts!

Wed, 27 May 2015 00:00:00 +0000

Introduction

In part 1 of this series we looked at the basics of core.async via channels and messages.
In part 2 we will explore timeouts and working with multiple channels using examples of calling out to web APIs.

We will be using the example of a web site that wants to display weather information and will be making mock calls to the OpenWeatherMap API and Weather Underground API to demonstrate using multiple channels with core.async

Helper Function

First we define a function that will be used repeatedly in the examples.

(defn set-inner-html!
  "Helper function to set contents of a DOM element."
  [elem value]
  (set! (.-innerHTML (goog.dom/getElement elem)) value))

It modifies DOM elements using the Google Closure Library.

Working with Timeouts

Let say that when a button is clicked we want to wait a second before continuing. Click the button below to see it in action.

Output:

The steps taken look like this:

In Clojure we could run in a different thread and use the Thread/sleep method provided by the Java virtual machine. But since ClojureScript uses a JavaScript runtime (which is single threaded) we don't have as straightforward of a solution.

One approach would be to call js/setTimeout and pass it a call back function. But here we will use the timeout function provided by core.async.

Given a html fragment like:


  Output:

When the button is clicked we can wait for 1 second and then make a DOM change like this:

(defn handle-timeout-button-click
  "Example of timeouts."
  [_]
  (set-inner-html! "timeout-output" "Button clicked, waiting...")
  (go
    (

timeout returns a channel that will close and will park until a value becomes available or the channel is closed.

Mock API Call Functions

The remaining examples will use these functions to simulate calling out to a couple APIs.

(defn mock-api-call "Wait ms milliseconds. Returns a channel containing the message" [ms msg] (go (

They take a random amount of time (up to 1.5 seconds) to return results. Note that a go block returns a channel which in this case will contain the message from the API.

Working with Multiple Channels via Parking

In this example we will demonstrate calling out to two APIs, waiting for both to return results, and then display both of them.

(defn handle-parking-button-click "Call two APIs. Output results from both." [_] (set-inner-html! "parking-output" "Calling APIs, waiting for results.") (go (let [weather-underground-chan (weather-underground-api-call) open-weather-map-chan (open-weather-map-api-call) weather-underground-message (

Output:

Under the hood the go block will transform its body to a state machine and wait until both of the calls to have received a value.

Working with Multiple Channels via alts!

In this example we will demonstrate calling out to two APIs, return results from the one that finishes the fastest, and then display it.

Since these are both weather services, in all likelihood they will return similar values, so we might do this if all we care about is getting a result and displaying it as fast as possible.

(defn handle-alts-button-click "Call two APIs. Output results from the first to return." [_] (set-inner-html! "alts-output" "Calling APIs, waiting for results.") (go (let [weather-underground-chan (weather-underground-api-call) open-weather-map-chan (open-weather-map-api-call) [msg] (alts! [weather-underground-chan open-weather-map-chan])] (set-inner-html! "alts-output" msg))))

First API to return results:

alts! will park until the first channel has a result. Since there is some randomness in the amount of time each API takes to respond in this example, click the button above a few times and we will potentially get results from a different one on multiple runs.

Tags: clojure clojurescript

Adventures in Clojure with core.async - Part 1 - Channels and Messages

Wed, 15 Apr 2015 00:00:00 +0000

Introduction

In part 1 of this series we will look at the basics of core.async via channels and messages.

In part 2 we explore timeouts and working with multiple channels using examples of calling out to web APIs.

core.async is a Clojure/ClojureScript library to facilitate asynchronous programming using channels.

I wanted to show some simple examples with live demos in the browser using Clojurescript, but if you would like to read a more thorough overview there is a great overview of core.async at the clojure.com blog. Code related to this post is also available at GitHub.

It builds upon the Communicating Sequential Processes (CSP) model described by Tony Hoare in the late 1970's, but seems to have seen a resurgence in popularity due to its usage in the Go programming language.

Sending Messages

These are some of the functions available for writing to and reading from a channel.

Function

put! Write a message asynchronously to a channel.

take! Take a message asynchronously from a channel.

>!! Write a message to a channel, blocks if no buffer space is available.

Read a message from a channel, blocks if nothing is available.

>! Write a message to a channel, parks if no buffer space is available. Must be called in a go block.

Read a message from a channel, parks if nothing is available. Must be called within a go block.

Note that >!! and

go Blocks

We can use the go macro provided by core.async to create new processes. It will turn any channel operations within into a state machine, which parks on a blocking operation, and resumes when the blocking operation completes.

The concept of parking will be covered in more depth in a future post.

Code

For the sake of completeness here is a helper function to set DOM values in the browser using the Google Closure Library.

(defn set-inner-html! "Helper function to set contents of a DOM element." [elem value] (set! (.-innerHTML (goog.dom/getElement elem)) value))

We will use in the examples below.

Unbuffered Channels

The simplest channel is unbuffered and created using the chan function with no arguments.

Writing to an unbuffered channel blocks further writes until the message is read by a consumer of the channel. So in this example we must read from and write to the channel in separate go blocks, which will allow them to appear to run concurrently.

(defn handle-example-zero-button-click "Write to and read from an unbuffered channel." [_] (let [ch (chan)] ;; Write to a channel (go (>! ch "Hello from an unbuffered channel!")) ;; Read from the channel (go (let [msg (

Unbuffered channel output:

Buffered Channels

Calling chan with a single argument that is a number creates a channel with a buffer. This allows us to write more than one message to the channel before it blocks. In the example below we create a channel with a buffer of 5.

Note that since this is not an unbuffered channel (and we are not writing more messages than the size of the buffer) we do not need to run the reads and writes in separate go blocks.

(defn handle-example-one-button-click "Write to and read from a buffered channel." [_] (go (let [ch (chan 5) _ (>! ch "Hello from a buffered channel!") msg (

Buffered channel output:

Dropping Buffers

We can also create a channel with a dropping-buffer. In this case writes will complete when the buffer is full, but the value will be dropped.

We are also using the onto-chan function here, which will write the contents of a collection onto a channel. In this case we are writing the elements of the list (0 1 2 3 4 5 6 7 8 9) to the channel. Note that onto-chan will also close the channel.

We are also using the specialized into function provided by core.async, which returns a channel containing a message consisting of the items in the channel conjoined to the collection passed in.

(defn handle-example-two-button-click "Write to and read from a channel with a dropping buffer." [_] (go (let [ch (chan (dropping-buffer 5)) _ (onto-chan ch (range 0 10)) msg (

Dropping buffer output:

Sliding Buffers

A sliding-buffer is similar to a dropping-buffer, except that when the buffer is full, writes will continue to succeed but the oldest element in the buffer will be dropped.

(defn handle-example-three-button-click "Write to and read from to a channel with a sliding buffer." [_] (go (let [ch (chan (sliding-buffer 5)) _ (onto-chan ch (range 0 10)) msg (

Sliding buffer output:

Conclusion

We've barely scratched the surface of core.async in this introductory post. In future posts we'll see more of what the library provides, show some practical examples (including some very neat things we can do with the state machine that go blocks provide).

In part 1 of this series we looked at the basics of core.async via channels and messages.

In [part 2][part-2] we will explore timeouts and working with multiple channels using examples of calling out to web APIs.

Tags: clojure clojurescript

Visualizing Twitter Connections with D3 and ClojureScript

Thu, 19 Feb 2015 00:00:00 +0000

Introduction

I was curious about the connections between some of the people I follow on Twitter and wanted to try visualizing it using D3 and ClojureScript using a force directed graph. The end result looks like this:

Not a lot of surprises, there are a few clusters around common interests: Emacs, food, Dwarf Fortress, Hadoop, etc.

Code

In this post i'll just share a minimum of ClojureScript code, but a code for a full example is available at Github. I highly recommend checking out the code, running the commands in the README file, and then viewing the results in a browser.

(ns force-directed-graph.core (:require cljsjs.d3)) (defn- build-force-layout [width height] (.. js/d3 -layout force (charge -140) (linkDistance 40) (size (array width height)))) (defn- setup-force-layout [force-layout graph] (.. force-layout (nodes (.-nodes graph)) (links (.-links graph)) start)) (defn- build-svg [width height] (.. js/d3 (select ".app") (append "svg") (attr "width" width) (attr "height" height))) (defn- build-links [svg graph] (.. svg (selectAll ".link") (data (.-links graph)) enter (append "line") (attr "class" "link") (attr "stroke" "grey") (style "stroke-width" 1))) (defn- build-nodes [svg graph force-layout] (.. svg (selectAll ".node") (data (.-nodes graph)) enter (append "text") (attr "cx" 12) (attr "cy" ".35em") (text #(.-name %)) (call (.-drag force-layout)))) (defn on-tick [link node] (fn [] (.. link (attr "x1" #(.. % -source -x)) (attr "y1" #(.. % -source -y)) (attr "x2" #(.. % -target -x)) (attr "y2" #(.. % -target -y))) (.. node (attr "transform" #(str "translate(" (.. % -x) "," (.. % -y) ")"))))) (defn ^:export main [json-file] (let [width 960 height 600 force-layout (build-force-layout width height) svg (build-svg width height)] (.json js/d3 json-file (fn [error json] (.. json -links (forEach #(do (aset %1 "weight" 1.0) (aset %1 "index" %2)))) (setup-force-layout force-layout json) (let [links (build-links svg json) nodes (build-nodes svg json force-layout)] (.on force-layout "tick" (on-tick links nodes)))))))

We are pulling in D3 via CLJSJS packages, which makes adding dependencies on Javascript libraries very easy. Otherwise, it is all pretty straightforward and not too different from an example like this in Javascript.

Sample input

The one thing to be aware of is how the JSON that will be passed in needs to be formatted. You will need to define nodes and links between the nodes.

Here is a simple example with four nodes, in this case the node named A follows B and C, while B follows D.

{ "nodes": [ { "name": "A", "index": 0 }, { "name": "B", "index": 1 }, { "name": "C", "index": 2 }, { "name": "D", "index": 3 } ], "links": [ { "target": 1, "source": 0 }, { "target": 2, "source": 0 }, { "target": 3, "source": 1 } ] }

Tags: d3 clojurescript

Writing a Kafka Producer and High Level Consumer in Clojure

Wed, 14 Jan 2015 00:00:00 +0000

Introduction

Kafka is a platform for handling real-time data feeds. In some ways it is like a database that exposes semantics of a messaging system.

The Kafka documentation provides an excellent overview which I have provided an extract from:

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. * Kafka maintains feeds of messages in categories called topics. * We'll call processes that publish messages to a Kafka topic producers. * We'll call processes that subscribe to topics and process the feed of published messages consumers. * Kafka is run as a cluster comprised of one or more servers each of which is called a broker.

In this post we'll use Clojure to write a producer that periodically writes random integers to a Kafka topic, and a High Level Consumer that reads them back. I am using Kafka 0.8.1.1 and assume it and ZooKeeper are running on localhost. A quickstart is available that can walk you through downloading and starting the services.

The source code for this project is available on GitHub.

Create a Project

We'll be using Leiningen to build and run our app.

Create a file called project.clj with the following contents:

(defproject hello-world-kafka "0.1.0" :description "Create a kafka producer and high level consumer" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.6.0"] [org.apache.kafka/kafka_2.9.2 "0.8.1.1" :exclusions [javax.jms/jms com.sun.jdmk/jmxtools com.sun.jmx/jmxri]]] :aot [hello-world-kafka.core] :main hello-world-kafka.core)

Next create a file /src/hello_world_kafka/core.clj with these contents:

(ns hello-world-kafka.core (:import (kafka.consumer Consumer ConsumerConfig KafkaStream) (kafka.producer KeyedMessage ProducerConfig) (kafka.javaapi.producer Producer) (java.util Properties) (java.util.concurrent Executors)) (:gen-class))

Producer Code

(defn- create-producer "Creates a producer that can be used to send a message to Kafka" [brokers] (let [props (Properties.)] (doto props (.put "metadata.broker.list" brokers) (.put "serializer.class" "kafka.serializer.StringEncoder") (.put "request.required.acks" "1")) (Producer. (ProducerConfig. props)))) (defn- send-to-producer "Send a string message to Kafka" [producer topic message] (let [data (KeyedMessage. topic nil message)] (.send producer data)))

Creating a producer and sending a message is pretty straightforward. Call the create-producer function with a list of Kafka brokers, and when you want to send a message pass the producer to the send-to-producer method along with the name of the topic and the message.

Consumer Code

(defrecord KafkaMessage [topic offset partition key value-bytes]) (defn- create-consumer-config "Returns a configuration for a Kafka client." [] (let [props (Properties.)] (doto props (.put "zookeeper.connect" "127.0.0.1:2181") (.put "group.id" "group1") (.put "zookeeper.session.timeout.ms" "400") (.put "zookeeper.sync.time.ms" "200") (.put "auto.commit.interval.ms" "1000")) (ConsumerConfig. props))) (defn- consume-messages "Continually consume messages from a Kafka topic and write message value to stdout." [stream thread-num] (let [it (.iterator ^KafkaStream stream)] (println (str "Starting thread " thread-num)) (while (.hasNext it) (as-> (.next it) msg (KafkaMessage. (.topic msg) (.offset msg) (.partition msg) (.key msg) (.message msg)) (println (str "Received on thread " thread-num ": " (String. (:value-bytes msg)))))) (println (str "Stopping thread " thread-num)))) (defn- start-consumer-threads "Start a thread for each stream." [thread-pool kafka-streams] (loop [streams kafka-streams index 0] (when (seq streams) (.submit thread-pool (cast Callable #(consume-messages (first streams) index))) (recur (rest streams) (inc index)))))

A few things to take note of:

The message is stored in Kafka as bytes, so in this case we need to turn the bytes into a String.

In consume-messages the call to the KafkaStream iterator .hasNext function is reading from a single partition of it's topic and will block until a message is received. So to read from multiple partitions we will need to run multiple threads of the consume-messages function.

The threads will run in a thread pool we later create with a java.util.concurrent.Executors

Clojure functions implement both Runnable and Callable, but since the executor's submit function is overloaded and can accept either, we must explicitly cast the function to a Callable.

Application code

(defn -main "Pull messages from a Kafka topic using the High Level Consumer" [topic num-threads] (let [consumer (Consumer/createJavaConsumerConnector (create-consumer-config)) consumer-map (.createMessageStreams consumer {topic (Integer/parseInt num-threads)}) kafka-streams (.get consumer-map topic) thread-pool (Executors/newFixedThreadPool (Integer/parseInt num-threads)) producer (create-producer "127.0.0.1:9092")] ;; Clean up on a SIGTERM or Ctrl-C (.addShutdownHook (Runtime/getRuntime) (Thread. #(do (.shutdown consumer) (.shutdown thread-pool)))) ;; Connect and start listening for messages on Kafka (start-consumer-threads thread-pool kafka-streams) ;; Send a random int to Kafka every 500 milliseconds (loop [] (let [num (str (rand-int 1000))] (println (str "Sending to Kafka topic " topic ": " num)) (send-to-producer producer topic num) (Thread/sleep 500) (recur)))))

The above code sets up our thread pools, creates and starts some consumers, and then sends a random integer between 0 and 999 to a topic every 500 milliseconds.

Running from the command line expects arguments like this:

$ lein trampoline run

We use lein trampoline so we can catch a SIGTERM or Control-C and clean up prior to shutting down.

If we want to send and read messages from a topic called random_numbers and use 2 threads for the consumers we can start the app like this:

$ lein trampoline run random_numbers 2

You should see output that looks something like this:

Starting thread 0 Starting thread 1 Sending to Kafka topic random_numbers: 753 Received on thread 1: 753 Sending to Kafka topic random_numbers: 971 Received on thread 1: 971 Sending to Kafka topic random_numbers: 56 Received on thread 1: 56 Sending to Kafka topic random_numbers: 536 Received on thread 1: 536 Sending to Kafka topic random_numbers: 589 Received on thread 1: 589 Stopping thread 0 Stopping thread 1

Tags: clojure kafka

Running a Dockerized Clojure Web App on CoreOS

Sat, 06 Dec 2014 00:00:00 +0000

Introduction

In the previous post we created a Clojure web service and ran it in a Docker container. Here we will deploy that container on a 3 node CoreOS cluster running in Vagrant on a local development machine.

CoreOS

CoreOS is a minimal version of Linux meant for large scale server deployments.

It has a very different model than a Debian/Red Hat/etc distribution. The OS runs on a read-only partition, and applications/services run in Docker containers on a read-write filesystem.

It's web page describes it as:

A new Linux distribution that has been rearchitected to provide features needed to run modern infrastructure stacks. The strategies and architectures that influence CoreOS allow companies like Google, Facebook and Twitter to run their services at scale with high resilience.

The components to be aware of are:

CoreOS - The underlying operating system.

Docker - CoreOS applications run in containers. At this time only Docker is supported. However, the CoreOS team recently announced a new container runtime called Rocket they will also be adding support for.

etcd - A highly-available key value store for shared configuration and service discovery. This fulfills a role similar to ZooKeeper.

fleet - You will use this to start/stop containers, define a cluster topology, etc. It ties together systemd and etcd into a distributed init system. Think of it as an extension of systemd that operates at the cluster level instead of the machine level.

We won't be really be going into etcd in this post, but there are some interesting things you can do with it. For instance, you could configure a trigger so that when a new web service comes online it can detect this and automatically change an nginx configuration to proxy traffic to it (or remove it from the rotation when the service is stopped).

Start Vagrant

I am assuming you are on either a Linux or OS X machine, if you are on Windows you will likely need to make a few changes.

If you run into any problems there is also a guide to using Vagrant on the CoreOS blog and documentation which may help you troubleshoot.

Ensure you have installed Vagrant on you development machine, then use git to clone the coreos-vagrant repo.

$ git clone git@github.com:coreos/coreos-vagrant.git $ cd coreos-vagrant

Rename the file config.rb.sample to config.rb and change the lines:

#$num_instances=1 #$update_channel='alpha'

to

$num_instances=3 $update_channel='stable'

Next rename the file user-data.sample to user-data and then browse to [[https://discovery.etcd.io/new]] and copy the what was on the page then uncomment and insert it on the the line that looks like this:

#discovery: https://discovery.etcd.io/

Every time you start a new cluster you will need a new token. If you do not want to use the discovery service you can instead include the network addresses of the machines that will be running etcd yourself. The CoreOS team provide the discovery.etcd.io service as a convenience to make it easier to bootstrap a cluster, but you are not required to use it.

Now run this command:

$ vagrant up

and Vagrant will launch 3 VirtualBox machines running CoreOS.

$ vagrant status Current machine states: core-01 running (virtualbox) core-02 running (virtualbox) core-03 running (virtualbox)

As you can see, there are 3 machines running.

SSH to your cluster

In order to forward your SSH session to other machines in the cluster you will need to use a key that is installed on all machines. In this case one from Vagrant will already be installed on the machines and we can add it like this:

$ ssh-add ~/.vagrant.d/insecure_private_key

Now SSH into one of the boxes with agent forwarding:

$ vagrant ssh core-01 -- -A

We'll use the fleetctl tool to control our cluster. First lets see that we can see all 3 machines:

$ fleetctl list-machines MACHINE IP METADATA c313e784... 172.17.8.102 - c3ddc4fd... 172.17.8.101 - d0f027da... 172.17.8.103 -

You should see something like the above. We haven't defined any metadata, but you could do something like tag a machine backed by a SSD drive, and then when you launch a service like PostgreSQL tell fleet that it should only run on a machine with that metadata.

Create A Service File

CoreOS runs applications as Docker containers. We will be running a simple web server written in Clojure that just displays the text "Hello World". The source code is available on GitHub and the Docker container is also published at Docker Hub.

Ensure that you are logged onto a machine in the cluster via SSH and create a file named clojure-http@.service. The file format is a systemd service file.

[Unit] Description=clojure-http [Service] ExecStart=/usr/bin/docker run --name clojure-http-%i --rm -p 5000:5000 -e PORT=5000 wtfleming/docker-compojure-hello-world ExecStop=/usr/bin/docker stop clojure-http-%i [X-Fleet] Conflicts=clojure-http@*.service

This file provides the instructions about how the service should run. There are a few things to note:

We are running a docker container and exposing port 5000.

The wtfleming/docker-compojure-hello-world container is publicly hosted on Docker Hub, it will be automatically downloaded to the CoreOS cluster.

The %i refers to the string between the @ and the suffix when we start the service (which we will do shortly, and should make sense then).

The X-Fleet directive indicates that this service should only one instance of the service on a machine. Since we've bound to port 5000 if we start up a second copy of this service we don't want it here, fleet will launch it on another machine in the cluster.

Now submit the service file and check that it registered:

$ fleetctl submit clojure-http@.service $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET clojure-http@.service aab3979 inactive inactive -

If you see the file listed we are ready to move on.

Start the Web Services

We will be running 2 copies of the service, lets start them:

$ fleetctl start clojure-http@1.service Unit clojure-http@1.service launched on c313e784.../172.17.8.102 $ fleetctl start clojure-http@2.service Unit clojure-http@2.service launched on c3ddc4fd.../172.17.8.101

You can check the status of running units:

$ fleetctl list-units UNIT MACHINE ACTIVE SUB clojure-http@1.service c313e784.../172.17.8.102 active running clojure-http@2.service c3ddc4fd.../172.17.8.101 active running

You can view logs like this:

$ fleetctl journal clojure-http@1.service Dec 03 04:44:48 core-02 systemd[1]: Starting clojure-http... Dec 03 04:44:48 core-02 systemd[1]: Started clojure-http. Dec 03 04:44:52 core-02 docker[711]: Listening on port 5000

Finally lets view each of the pages:

$ curl http://172.17.8.101:5000 Hello World $ curl http://172.17.8.102:5000 Hello World

Hopefully you'll have gotten output similar to above.

Stop the Services and Clean Up

Lets stop the services:

$ fleetctl stop clojure-http@1.service Unit clojure-http@1.service loaded on c313e784.../172.17.8.102 $ fleetctl stop clojure-http@2.service Unit clojure-http@2.service loaded on c3ddc4fd.../172.17.8.101

Notice that they have now entered a failed state, but remain listed.

$ fleetctl list-units UNIT MACHINE ACTIVE SUB clojure-http@1.service c313e784.../172.17.8.102 failed failed clojure-http@2.service c3ddc4fd.../172.17.8.101 failed failed

We could restart them, but instead we will remove them like this:

$ fleetctl destroy clojure-http@1.service Destroyed clojure-http@1.service $ fleetctl destroy clojure-http@2.service Destroyed clojure-http@2.service $ fleetctl list-units UNIT MACHINE ACTIVE SUB

Now lets remove our service file.

$ fleetctl destroy clojure-http@.service Destroyed clojure-http@.service $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET

Now exit the cluster, shut it down, and clean up Vagrant.

$ exit $ vagrant halt ==> core-03: Attempting graceful shutdown of VM... ==> core-02: Attempting graceful shutdown of VM... ==> core-01: Attempting graceful shutdown of VM... $ vagrant destroy ...

And we're done!

Next Steps

So far we created a CoreOS cluster and ran a web server on it, but we have just barely scratched the surface of what is possible.

In a production environment you would want expose the web services to the world. If you are on EC2 you might want to register the services with an Elastic Load Balancer. One way to do that is with a "sidekick" container for each web service.

The CoreOS team provides code to create a container on github that will register a service with an ELB, and example of how to use it.

Marcel de Graaf has also written about running on EC2 and using nginx to proxy. If you want to know more, I highly recommend taking the time to read his post and the CoreOS documentation, they both go into much greater detail about what is possible and issues you should be aware of running a service in a production setting.

Tags: clojure docker coreos

Dockerizing a Clojure, Compojure, and HTTP Kit Web Application

Sat, 29 Nov 2014 00:00:00 +0000

Docker is "an open platform for developers and sysadmins to build, ship, and run distributed applications. With Docker, developers can build any app in any language using any toolchain. Dockerized apps are completely portable and can run anywhere - colleagues' OS X and Windows laptops, QA servers running Ubuntu in the cloud, and production data center VMs running Red Hat."

Here we will build a very simple web site using the Clojure Compojure and HTTP Kit libraries. The source code for this article is also available here.

Create a Compojure Application

First create a file named project.clj with the following contents:

(defproject hello-world "0.1.0" :description "Compojure hello world web app" :url "https://github.com/wtfleming/docker-compojure-hello-world" :min-lein-version "2.0.0" :dependencies [[org.clojure/clojure "1.6.0"] [compojure "1.2.1"] [http-kit "2.1.16"]] :main hello-world.core :aot [hello-world.core])

And then create a file named src/hello_world/core.clj with these contents:

(ns hello-world.core (:require [compojure.core :refer :all] [org.httpkit.server :refer [run-server]]) (:gen-class)) (defroutes myapp (GET "/" [] "Hello World")) (defn -main [] (let [port (Integer/parseInt (or (System/getenv "PORT") "8080"))] (run-server myapp {:port port}) (println (str "Listening on port " port))))

This is a very basic Compojure/HTTP Kit app with a single route that displays a web page with the contents "Hello World". We also check to see if an environment variable PORT has been defined, and if so bind to it, otherwise default to port 8080.

Now build with Leiningen.

$ lein uberjar

You could now run locally like this

$ java -jar target/hello-world-0.1.0-standalone.jar

Browse to http://localhost:8080 and you should see a page that displays "Hello World".

Create a Dockerfile

Now create a file called Dockerfile with these contents

# # Dockerfile for a compojure hello world app # FROM java:openjdk-8-jre MAINTAINER Will Fleming ENV REFRESHED_AT 2014-11-25 COPY target/hello-world-0.1.0-standalone.jar hello-world-0.1.0-standalone.jar ENV PORT 4000 EXPOSE 4000 CMD ["java", "-jar", "hello-world-0.1.0-standalone.jar"]

The Dockerfile is a set of instructions to build a docker container.

We copy the jar file from our file system to the docker filesystem.

Define a environment variable PORT with the value 4000.

Expose port 4000 to other docker containers.

Provide a default command to be run when starting the container.

There is much more you can do, the Dockerfile documentation goes into further detail.

Build a Docker Image

Now lets build a Docker image.

$ docker build -t wtfleming/compojure-hello-world .

We can see that it was built:

$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE wtfleming/compojure-hello-world latest b1c798937b9d 6 days ago 351.1 MB

We can use an image as the basis to launch a container.

Run a Docker container

$ docker run --rm -p 4000:4000 wtfleming/compojure-hello-world Listening on port 4000

In the above we:

Start a container from the wtfleming/compojure-hello-world image.

–rm will delete the container when it stops

-p will publish a container's port to the host

Open a web browser to http://localhost:4000 and you will see the page.

Browse to http://localhost:4000 or if you are on a Mac or Windows using boot2docker find the ip address like

$ boot2docker ip The VM's Host only interface IP address is: 192.168.59.103

and browse to the ip provided by boot2docker like http://192.168.59.103:4000

Tags: clojure docker

Geospatial queries with elasticsearch

Fri, 24 Oct 2014 00:00:00 +0000

Recently I was investigating the geospatial capabilities of elasticsearch for a project (in this case I am using version 1.3.4). While I ultimately ended up not using it for this project I thought I would write up an example using the command line.

Create a mapping

First we need to create a mapping for the geotest index. A mapping allows us to tell elasticsearch how an index is structured. In this case we expect our documents to have two fields: a user identifier, and a location (in this case we are using a 2 dimensional point to represent lat/lon).

curl -XPUT http://localhost:9200/geotest -d ' { "mappings": { "usergeo": { "properties": { "udid": {"type": "string"}, "location": {"type": "geo_point"} } } } } '

Add users to the index

curl -XPOST 'http://localhost:9200/geotest/usergeo' -d ' { "udid" : "foo-udid", "location": {"lat": "38.897676", "lon": "-77.03653"} } ' curl -XPOST 'http://localhost:9200/geotest/usergeo' -d ' { "udid" : "bar-udid", "location": {"lat": "28.897676", "lon": "-81.03653"} } '

Run a query

Find all users located within a polygon.

curl -XGET 'http://localhost:9200/geotest/usergeo/_search?pretty=true' -d ' { "query": { "filtered" : { "query" : { "match_all" : {} }, "filter" : { "geo_polygon" : { "location" : { "points" : [ {"lat" : 30, "lon" : -70}, {"lat" : 30, "lon" : -80}, {"lat" : 40, "lon" : -80}, {"lat" : 40, "lon" : -70} ] } } } } } } '

You should see output similar to the following. As you can see, elasticsearch only returned the one user that was within the polygon.

{ "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "geotest", "_type" : "usergeo", "_id" : "_R-YQd-eS-2vc3rBzhibAQ", "_score" : 1.0, "_source": { "udid" : "foobar-udid", "location": {"lat": "38.897676", "lon": "-77.03653"} } } ] } }

Tags: elasticsearch

Clojure on the BeagleBone part 4 - Digital Input with ClojureScript

Thu, 23 Oct 2014 00:00:00 +0000

Intro

In part 1 of this series we saw how to install Java and Leiningen on a BeagleBone.

In part 2 we used the BeagleBone to blink an LED on a breadboard using Clojure.

In part 3 we blinked an LED using ClojureScript.

In part 4 we read digital inputs via polling and interrupts.

The BeagleBone is a 1GHz ARM board with 512Mb of RAM capable of running Linux. The Debian image ships with node.js and a javascript library in [Bonescript][bonescript] for interacting with the hardware. Clojurescript provides a compiler for Clojure that targets Javascript, here we will use it to and two methods of reading digital input and light an LED on a breadboard using the BeagleBone GPIO.

Reading from digital input via polling

Wire up the LED circuit

Using a jumper wire connect Pin 1 on Header P9 (ground) on the BeagleBone to the ground rail on the breadboard.

Place an LED in the breadboard.

Using a jumper wire connect the cathode (shorter) wire of the LED to the negative rail.

Connect one end of a 100 ohm resistor to the anode (longer) wire of the LED.

Using another jumper wire connect the other end of the resistor to Pin 14 on Header P8

Wire up the push button circuit

Using a jumper wire connect Pin 3 on Header P9 (3.3 volts) on the BeagleBone to the positive rail on the breadboard.

Place a push button in the breadboard.

Using a jumper wire connect one lead of the push button to the positive rail.

Connect a 10K ohm pull down resistor to the negative rail and the button's other lead.

Using a jumper wire connect Pin 18 on Header P9 to the same button lead we connected the 10k resistor in step four.

Note: The GPIO pins can only handle 3.3 volts, so be very careful that you do not accidentally connect a jumper to one of the 5 volt source pins. If you are unsure of what you are doing I would highly recommend reading the system reference manual to make sure you do not damage your board.

ClojureScript code

SSH into your BeagleBone and create a file named project.clj with the following contents:

(defproject cljs-beaglebone-digital-input "0.0.1" :description "Demonstrates reading digital inputs on a beaglebone" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :plugins [[lein-cljsbuild "1.0.3"]] :cljsbuild { :builds [{ :source-paths ["src"] :compiler { :output-to "lib/digital-input.js" :optimizations :simple :target :nodejs :pretty-print true}}]} :dependencies [[org.clojure/clojurescript "0.0-2197"]])

And create a file name src/cljs_digital_input/core.cljs with these contents:

(ns cljs-digital-input.core (:require [cljs.nodejs :as nodejs])) (def bone (nodejs/require "bonescript")) ; Bonescript identifiers related to pins (def OUTPUT (aget bone "OUTPUT")) (def INPUT (aget bone "INPUT")) ; Pins (def led-pin "P8_14") ; GPIO pin 14 on header P8 (def digital-input-pin "P8_18") ; GPIO pin 18 on header P8 (defn setup-pins! [] (.pinMode bone led-pin OUTPUT) (.pinMode bone digital-input-pin INPUT)) (defn read-input [] "Reads state of the digital input pin then turns the LED on or off" (let [input-pin-state (.digitalRead bone digital-input-pin)] (.digitalWrite bone led-pin input-pin-state))) (defn -main [& args] (setup-pins!) (js/setInterval read-input 1)) ; poll every millisecond (set! *main-cli-fn* -main)

Now compile the clojurescript to javascript and run it on your BeagleBone.

$ lein cljsbuild once $ node lib/digital-input.js

When you press the push button the LED should turn on, and when you release it will turn off.

Reading from digital input via interrupts

In the previous example we polled a pin once a millisecond to determine whether or not the push button was being pushed, an alternative is to use interrupts. We will use the same circuit that we wired up in the previous example. We now can be notified and react to an event (a button being pressed in this case) rather than continually having to check to see if something has happened. We use the [attachInterupt()][attach-interrupt] function bonescript provides.

On your BeagleBone and create a new directory and a file named project.clj with the following contents:

(defproject cljs-beaglebone-digital-input-interrupts "0.0.1" :description "Demonstrates reading digital inputs on a beaglebone" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :plugins [[lein-cljsbuild "1.0.3"]] :cljsbuild { :builds [{ :source-paths ["src"] :compiler { :output-to "lib/digital-input-interrupts.js" :optimizations :simple :target :nodejs :pretty-print true}}]} :dependencies [[org.clojure/clojurescript "0.0-2197"]])

And create a file name src/cljs_digital_input_interrupts/core.cljs with these contents:

(ns cljs-digital-input-interrupts.core (:require [cljs.nodejs :as nodejs])) (def bone (nodejs/require "bonescript")) ; Bonescript identifiers related to pins (def OUTPUT (aget bone "OUTPUT")) (def INPUT (aget bone "INPUT")) (def HIGH (aget bone "HIGH")) (def LOW (aget bone "LOW")) (def CHANGE (aget bone "CHANGE")) ; Pins (def led-pin "P8_14") ; GPIO pin 14 on header P8 (def digital-input-pin "P8_18") ; GPIO pin 18 on header P8 (defn setup-pins! [] (.pinMode bone led-pin OUTPUT) (.pinMode bone digital-input-pin INPUT)) (defn button-callback [x] (if (= (.-value x) HIGH) (.digitalWrite bone led-pin HIGH) (.digitalWrite bone led-pin LOW))) (defn do-nothing [] "Does nothing" (do)) (defn -main [& args] (setup-pins!) (.attachInterrupt bone digital-input-pin true CHANGE button-callback) ; Endlessly loop doing nothing but prevents node.js from exiting (js/setInterval do-nothing 1000)) (set! *main-cli-fn* -main)

Now compile the clojurescript to javascript and run it on your BeagleBone.

$ lein cljsbuild once $ node lib/digital-input-interrupts.js

Once again, when you press the push button the LED should turn on, and when you release it will turn off.

Tags: clojure beaglebone

Clojure on the BeagleBone part 3 - Blinking an LED with ClojureScript

Mon, 14 Jul 2014 00:00:00 +0000

Intro

In part 1 of this series we saw how to install Java and Leiningen on a BeagleBone.

In part 2 we used the BeagleBone to blink an LED on a breadboard using Clojure.

In part 3 we will blink an LED using ClojureScript.

In part 4 we will read digital inputs via polling and interrupts.

The BeagleBone is a 1GHz ARM board with 512Mb of RAM capable of running Linux. The Debian image ships with node.js and a javascript library in BoneScript for interacting with the hardware. Clojurescript provides a compiler for Clojure that targets Javascript, here we will use it to blink an LED on a breadboard using the BeagleBone GPIO.

Hello World with Clojurescript and node.js

First we will ensure that ClojureScript and node.js are setup. Ensure you have followed the instructions in the first part of this series to install Leiningen.

SSH into your BeagleBone and create a file named project.clj with the following contents:

(defproject cljs-nodejs-hello-world "0.0.1" :description "Hello World example for clojurescript on node.js" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :plugins [[lein-cljsbuild "1.0.3"]] :cljsbuild { :builds [{ :source-paths ["src"] :compiler { :output-to "lib/hello-world.js" :optimizations :simple :target :nodejs :pretty-print true}}]} :dependencies [[org.clojure/clojurescript "0.0-2197"]])

And create a file name src/cljs_helloworld/core.cljs with these contents:

(ns cljs-helloworld.core (:require [cljs.nodejs :as nodejs])) (defn -main [& args] (println (apply str (map [\space "world" "hello"] [2 0 1])))) ; Let println work (nodejs/enable-util-print!) ; Tell node which function to use as main (set! *main-cli-fn* -main)

Now compile the clojurescript to javascript and run it on your BeagleBone.

$ lein cljsbuild once $ node lib/hello-world.js

You should see hello world output.

Blinking an LED

This is inspired by an example from the excellent book [Getting Started With BeagleBone][beaglebone-book] by [Matt Richardson][matt-richardson].

Schematic above was generated using the open source Fritzing.

Note: The GPIO pins can only handle 3.3 volts, so be very careful that you do not accidentally connect a jumper to one of the 5 volt source pins. If you are unsure of what you are doing I would highly recommend reading the system reference manual to make sure you do not damage your board.

Wire up the LED

Using a jumper wire connect Pin 2 on Header P9 (ground) on the BeagleBone to the negative rail on the breadboard.

Place an LED in the breadboard.

Using a jumper wire connect the cathode (shorter) wire of the LED to the negative rail.

Connect one end of a 100 ohm resistor to the anode (longer) wire of the LED.

Using another jumper wire connect the other end of the resistor to Pin 13 on Header P8

Now we are ready to write the code.

ClojureScript code

SSH into your BeagleBone and create a file named project.clj with the following contents:

(defproject cljs-beaglebone-blink-led "0.0.1" :description "Hello World example for clojurescript on node.js" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :plugins [[lein-cljsbuild "1.0.3"]] :cljsbuild { :builds [{ :source-paths ["src"] :compiler { :output-to "lib/blink.js" :optimizations :simple :target :nodejs :pretty-print true}}]} :dependencies [[org.clojure/clojurescript "0.0-2197"]])

And create a file named src/cljs_blink_led/core.cljs with these contents:

(ns cljs-blink-led.core (:require [cljs.nodejs :as nodejs])) (def bone (nodejs/require "bonescript")) ; Bonescript identifiers related to pins (def OUTPUT (aget bone "OUTPUT")) (def HIGH (aget bone "HIGH")) (def LOW (aget bone "LOW")) ; LEDs (def led-pin "P8_13") ; GPIO pin 13 on header P8 (def onboard-led "USR3") ; One of the onboard LEDs ; State of our LEDs (def led-pin-state (atom LOW)) ; Set both pins as an output (defn setup-pins [] (.pinMode bone led-pin OUTPUT) (.pinMode bone onboard-led OUTPUT)) (defn toggle-digital-pin-state [] (if (= @led-pin-state HIGH) (reset! led-pin-state LOW) (reset! led-pin-state HIGH))) (defn blink-leds [] (toggle-digital-pin-state) (.digitalWrite bone led-pin @led-pin-state) (.digitalWrite bone onboard-led @led-pin-state)) (defn -main [& args] (setup-pins) (js/setInterval blink-leds 1000)) (set! *main-cli-fn* -main)

Now compile the clojurescript to javascript and run it on your BeagleBone.

$ lein cljsbuild once $ node lib/blink.js

Your LED should now blink on and off every second.

Tags: clojure beaglebone

Clojure on the BeagleBone part 2 - Blinking an LED with Clojure

Sun, 25 May 2014 00:00:00 +0000

Intro

In part 1 of this series we saw how to install Java and Leiningen on a BeagleBone.

In part 2 we use the BeagleBone to blink an LED on a breadboard using Clojure.

In part 3 we will blink an LED using ClojureScript.

In part 4 we will read digital inputs via polling and interrupts.

This is inspired by an example from the excellent book Getting Started With BeagleBone by Matt Richardson.

Schematic above was generated using the open source Fritzing.

Note: The GPIO pins can only handle 3.3 volts, so be very careful that you do not accidentally connect a jumper to one of the 5 volt source pins. If you are unsure of what you are doing I would highly recommend reading the system reference manual to make sure you do not damage your board.

Wire up the LED

Using a jumper wire connect Pin 2 on Header P9 (ground) on the BeagleBone to the negative rail on the breadboard.

Place an LED in the breadboard.

Using a jumper wire connect the cathode (shorter) wire of the LED to the negative rail.

Connect one end of a 100 ohm resistor to the anode (longer) wire of the LED.

Using another jumper wire connect the other end of the resistor to Pin 13 on Header P8

Clojure code

SSH into your BeagleBone and create a file named project.clj with the following contents:

(defproject clj-blink-led "0.1.0" :description "Blink a LED on a BeagleBone" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.6.0"]] :main clj-blink-led.core)

And create a file name src/clj_blink_led/core.clj with these contents:

(ns clj-blink-led.core (:require [clojure.java.io :as io])) (defn write-to-filesystem "Helper function for setting pins" [location val] (with-open [f (io/output-stream location)] (.write f (.getBytes val)))) (defn cleanup "Tidy up when a user presses Ctrl-C to exit" [] (println "Shutting down...") (write-to-filesystem "/sys/class/gpio/unexport" "23")) (defn setup "Export pin and set the direction" [] (.addShutdownHook (Runtime/getRuntime) (Thread. cleanup)) (write-to-filesystem "/sys/class/gpio/export" "23") (write-to-filesystem "/sys/class/gpio/gpio23/direction" "out")) ; Pin states (def HIGH "1") (def LOW "0") (defn -main [] (println "Blinking LED, press Ctrl-C to exit") (setup) (loop [pin-state HIGH] (write-to-filesystem "/sys/class/gpio/gpio23/value" pin-state) (Thread/sleep 1000) (recur (if (= pin-state HIGH) LOW HIGH))))

Now run it on your BeagleBone as the root user.

$ lein run

Your LED should now blink on and off every second.

Tags: clojure beaglebone

Clojure on the BeagleBone part 1 - Installing Java, Leiningen, and Emacs 24

Sat, 24 May 2014 00:00:00 +0000

Intro

In part 1 of this series we see how to install Java and Leiningen on a BeagleBone.

In part 2 we use the BeagleBone to blink an LED on a breadboard using Clojure.

In part 3 we will blink an LED using ClojureScript.

In part 4 we will read digital inputs via polling and interrupts.

The BeagleBone is a small and low cost ($55) 1Ghz ARM board with 512Mb of RAM capable of running Linux.

These instructions assume that you are using Debian 2014-05-14 as the root user. You can obtain the latest versions of Debian and Angstrom here.

If you are using a different operating system or version you may need to make some changes.

Java

I started with the BeagleBone Oracle JDK instructions, but needed to make some changes (for instance because I am using Debian I can use "hard float" JDK - however if you are using Angstrom Linux you may need the "soft float" version - or you may want to use Java 8 instead).

SSH into your BeagleBone and do the following:

# You must accept the Oracle Binary Code License Agreement for Java SE. # The license and download links are available at: # http://www.oracle.com/technetwork/java/javase/downloads/jdk7-arm-downloads-2187468.html # You can accept the license via wget like this: $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-arm-vfp-hflt.tar.gz $ tar xzf jdk-7u55-linux-arm-vfp-hflt.tar.gz $ export PATH=$PATH:/root/jdk1.7.0_55/bin $ export JAVA_HOME=/root/jdk1.7.0_55 $ java -version java version "1.7.0_55" Java(TM) SE Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) Client VM (build 24.55-b03, mixed mode)

You will want to store all the exports from this post in your .bashrc so you do not need to type them in every time you log in.

Leiningen

Install following the installation instructions here:

$ mkdir bin $ cd bin $ wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein $ chmod 755 lein $ export PATH=$PATH:/root/bin # Generally you do not want to run as root, but since we will want access # to the GPIO pins it is easier to develop on a BeagleBone as root. # Turn off leiningen's warning about running as root. $ export LEIN_ROOT=1

Now lets fire up a REPL and make sure it works.

$ lein repl user=> (println "Hello BeagleBone!") Hello BeagleBone nil

Emacs 24.4

The latest version of Emacs in the Debian package list is 23, here is how to download and install 24.4. Note that generally you will be happier running Emacs on you laptop/desktop and using tramp-mode to edit files from there.

Run the following on your BeagleBone and go get a cup of coffee, it will take at least an hour.

$ apt-get install build-essential libncurses5-dev $ wget http://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz $ tar xzvf emacs-24.4.tar.gz $ cd emacs-24.4 $ ./configure --with-xpm=no --with-gif=no --without-x $ make $ make install

Tags: clojure beaglebone

Blinking an LED with the MSP430

Wed, 06 Nov 2013 00:00:00 +0000

I recently bought a MSP430 Launchpad from Texas Instruments. The MSP430 is a family of low power, low cost microcontrollers. To make sure my development environment was setup correctly I wanted to start with one of the simplest things to do, blink a LED.

I have a wire connected to P1.0, another connected to GND, a 180 ohm resistor, and an LED 180 Ohm resistor. The code is compiled with GCC.

blink.c

#include int main(void) { WDTCTL = WDTPW + WDTHOLD; // Stop watchdog timer P1DIR = 0x01; // Set pin to output, 0b00000001 for (;;) { volatile unsigned int i; P1OUT ^= 0x01; // toggle LED1 (P1.0) on/off, 0b00000001 i = 50000; do (i--); while (i != 0); } }

Makefile

CC=msp430-gcc CFLAGS=-Os -Wall -g -mmcu=msp430g2553 OBJS=main.o all: $(OBJS) $(CC) $(CFLAGS) -o main.elf $(OBJS) %.o: %.c $(CC) $(CFLAGS) -c $< clean: rm -fr main.elf $(OBJS)

We build, then use mspdebug to load the program onto the chip from the command line.

$ make $ mspdebug rf2500 (mspdebug) prog main.elf Erasing... Programming... Writing 108 bytes at c000 [section: .text]... Writing 32 bytes at ffe0 [section: .vectors]... Done, 140 bytes total (mspdebug) exit

Now we have a LED blinking.

Tags: msp430

Querying Stack Exchange database dumps with Cascalog

Sun, 11 Aug 2013 00:00:00 +0000

Cascalog is a fully-featured data processing and querying library for Clojure. The main use cases for Cascalog are processing Big Data on top of Hadoop or doing analysis on your local computer from the Clojure REPL. Cascalog is a replacement for tools like Pig, Hive, and Cascading.

We will use Cascalog to run some basic queries on the Arqade Stack Exchange site database dump. Arqade is a website dedicated to video game questions and answers.

Every 3 months the team at Stack Exchange provides an data dump of all creative commons licensed questions and answers from network of websites (the largest of which being Stack Overflow.

Getting Started

The code and data for this post is available here. The relevant file is queries.clj.

I encourage you to follow along in a REPL. Run the commands that begin with:

user=>

Fire up a leiningen REPL in the project's directory and switch to the demo namespace:

user=> (use 'cascalog_stack_exchange.queries)

At the start of the queries.clj we have pulled in the following:

(ns cascalog_stack_exchange.queries (:use cascalog.api) (:require [cascalog.ops :as ops] [clojure.data.xml :as xml] [clojure.string :as str]))

This provides everything we will need to run the queries.

Parsing the XML

In queries.clj we have defined a function that will parse a line of XML representing a user, and extract the user id, display name, and reputation.

(defmapop user-xml-parser [user-xml] "Parse a line of xml representing a stack exchange user." (try (let [user (xml/parse-str user-xml) {{:keys [Id DisplayName Reputation]} :attrs} user] [Id DisplayName (Integer/parseInt Reputation)]) (catch Exception _ [nil nil nil])))

It takes input that looks like this

and returns

["3" "David Fullerton" 3272]

Querying Users

Our first query will iterate over all lines in the file storing user information. Extract their id, reputation score. And finally output the results to standard out.

(defn user-query "Run a query that outputs user id, reputation, and display name." [] (let [file-tap (hfs-textline "data/users.xml")] (?<- (stdout) [?id ?reputation ?display-name] (file-tap ?line) (user-xml-parser ?line :> ?id ?display-name ?reputation))))

Indicate that this is a Cascalog query which will run immediately.

?<-

Specify that the results will written to standard out. It is possible to store the output in any format and wherever you want (HDFS, a relational database, Amazon S3, etc.). But for simplicity we will just be using stdout in these examples.

(stdout)

Tell Cascalog that we want to output the ?id ?reputation? and ?display-name variables.

[?id ?reputation ?display-name]

We earlier defined file-tap as a generator for each line of text in the file data/users.xml. We put each of them in ?line variable.

(file-tap ?line)

Parse each line and use the :> keyword to bind them to the variables ?id, ?display-name, and ?reputation.

(user-xml-parser ?line :> ?id ?display-name ?reputation)

Now lets run the query, and look at the output:

user=> (user-query) RESULTS ----------------------- 3 3272 David Fullerton 4 101 Robert Cartaino 5 238 Jin -----------------------

Query with a filter

We will now run a similar query, but this time we want to only include users with more than 200 reputation.

(defn user-minimum-reputation-query "Run a query showing users with a reputation greater than 200." [] (let [file-tap (hfs-textline "data/users.xml")] (?<- (stdout) [?display-name ?reputation] (file-tap ?line) (user-xml-parser ?line :> _ ?display-name ?reputation) (> ?reputation 200))))

Running gives us:

user=> (user-minimum-reputation-query) RESULTS ----------------------- David Fullerton 3272 Jin 238 -----------------------

Counting with built in operations

Cascalog provides a number of built in operations. Here we want to know the total number of users with a reputation greater than 200.

(defn user-minimum-reputation-count-query "Run a query counting the number of users with a reputation greater than 200." [] (let [file-tap (hfs-textline "data/users.xml")] (?<- (stdout) [?count] (file-tap ?line) (user-xml-parser ?line :> _ ?display-name ?reputation) (> ?reputation 200) (ops/count :> ?count))))

Everything looks good:

user=> (user-minimum-reputation-count-query) RESULTS ----------------------- 2 -----------------------

Querying Posts

Once again we need to define a function to parse XML, this time for a post (a question or answer).

(defmapop post-xml-parser [post-xml] "Parse a line of xml representing a stack exchange post." (try (let [post (xml/parse-str post-xml) {{:keys [OwnerUserId PostTypeId Tags]} :attrs} post] [OwnerUserId PostTypeId Tags]) (catch Exception _ [nil nil nil])))

The following query is very similar to the user-query above. But note that in this case we are using !tags rather than ?tags. Variables beginning with a ? are non-nullable, Cascalog will filter out any records in which a non-nullable is bound to null.

Variables beginning with ! are nullable and will not be filtered out. Here we are demonstrating that posts with a post-type-id are answers, and do not contain tags. If you wanted to know which tags are associated with the answer it is possible to join on the answer's parent id.

(defn post-query "Run a query that outputs post owner-id, type-id, and tags" [] (let [file-tap (hfs-textline "data/posts.xml")] (?<- (stdout) [?owner-user-id ?post-type-id !tags] (file-tap ?line) (post-xml-parser ?line :> ?owner-user-id ?post-type-id !tags))))

Running the query this is what we should see:

user=> (post-query) RESULTS ----------------------- 4 1 3 1 3 1 4 2 null 5 1 4 2 null -----------------------

Aggregating Tags

If we wanted to see all the tags used by each user we can run a query like this.

(defbufferop aggregate-tags "Function to combine string containing tags" [tuples] [(reduce str (map first tuples))]) (defn post-aggregate-query "Run a query to get a list of all tags by each user" [] (let [file-tap (hfs-textline "data/posts.xml")] (?<- (stdout) [?owner-user-id ?tags] (file-tap ?line) (post-xml-parser ?line :> ?owner-user-id _ ?tags1) (aggregate-tags ?tags1 :> ?tags))))

Which will output:

user=> (post-aggregate-query) RESULTS ----------------------- 3 4 5 -----------------------

Joining users and posts

We will now combine what we have learnt to create a query that will output the display name and list of tags used by users with more than 200 reputation.

(defn user-tags-join-query "Run a query on users and posts that joins on user id and output user display name and tags by all users with more than 200 reputation." [] (let [users-tap (hfs-textline "data/users.xml") posts-tap (hfs-textline "data/posts.xml")] (?<- (stdout) [?display-name ?tags] (posts-tap ?posts-line) (post-xml-parser ?posts-line :> ?user-id _ ?raw-tags) (aggregate-tags ?raw-tags :> ?tags) (users-tap ?users-line) (user-xml-parser ?users-line :> ?user-id ?display-name ?reputation) (> ?reputation 200))))

Since display name is stored in users.xml and the questions are in posts.xml, we will need to join on user id, since that is in both files.

In the following code posts-line and users-line are from two different sources of data. Since they are both binding to a variable ?user-id, behind the scenes Cascalog will using a join to resolve the query.

(post-xml-parser ?posts-line :> ?user-id _ ?raw-tags) (user-xml-parser ?users-line :> ?user-id ?display-name ?reputation)

We can now run the query, and everything looks good:

user=> (user-tags-join-query) RESULTS ----------------------- David Fullerton Jin -----------------------

Tags: cascalog

Function
put!	Write a message asynchronously to a channel.
take!	Take a message asynchronously from a channel.
>!!	Write a message to a channel, blocks if no buffer space is available.
	Read a message from a channel, blocks if nothing is available.
>!	Write a message to a channel, parks if no buffer space is available. Must be called in a go block.
	Read a message from a channel, parks if nothing is available. Must be called within a go block.