nbeloglazov blog

Random Vector Generation

Sun, 09 Apr 2017 00:00:00 -0700

In this article, I want to discuss unit vector generation. It was inspired by the recent addition of random-2d and random-3d functions in Quil API. The goal is simple: write a function that generates a random unit vector, both in 2D and 3D. This article doesn’t use advanced math (e.g. statistics) and discusses the matter from software engineer point of view who forgot college course about statistics and theory around generating random values.

2D vectors

Let's start with the 2D. Here the naive approach that I would try first:

Generate x and y as random numbers from [-1, 1].
Normalize vector: divide x and y by the vector length to make it a unit vector.

Clojure implementation:

(defn rand-2d []
  (let [x (dec (rand 2))
        y (dec (rand 2))
        len (Math/sqrt (+ (* x x) (* y y)))]
    [(/ x len) (/ y len)]))

Now let’s see how well it works in practice. To do that let’s generate 2000 vectors and draw them. When drawing I "cut" beginning of vectors to make image cleaner:

It's not obvious, but you can see that there are more vectors in the "corners" at 45°, 135°, 225° and 315° and fewer vectors at 0°, 90°, 180° and 270°. You can start animation to see better. This clustering doesn’t seem good. Good vector generator must generate uniformly distributed vectors.

Let's try another approach that uses the fact that unit vector is "almost" defined by just one of its coordinate, x or y. Given x we can calculate y as ±√(1 - x²). So let’s try this approach: 1. Generate x as random number from [-1, 1]. 2. Calculate y as sqrt(1 - x * x). 3. Multiply y by -1 with 0.5 chance otherwise, y will be always positive.

Clojure implementation:

(defn rand-2d []
  (let [x (dec (rand 2))
        sign (rand-nth [-1 1])
        y (* sign (Math/sqrt (- 1 (* x x))))]
    [x y]))

Visualization:

Turns out it's even worse than the first approach. At least now we can clearly see that it's bad.

Let's try yet another approach. In the first and second approaches we saw vectors clustered around certain angles. So we need to make angles uniformly distributed. To achieve that instead of generating x and y we’ll randomly choose an angle and calculate x and y based on the angle. In other words we'll generate a unit vector in polar coordinates instead of cartesian. In cartesian coordinates vector is defined by a pair [x, y] while in polar the pair is [r, ϕ] where r is the vector length and ϕ is the angle:

Third (polar) approach:

Set r = 1 (unit vector). Generate ϕ as random number from [0, 2π]
Calculate x = r*cos(ϕ), y = r*sin(ϕ)

Clojure implementation:

(defn rand-2d []
  (let [phi (rand (* 2 Math/PI))]
    [(Math/cos phi)
     (Math/sin phi)]))

Visualization:

Finally it looks good. Comparing with the first approach vectors feel more uniformly distributed.

3D vectors

Great, we figured out 2D. What about 3D then? The situation is very similar. Let’s try the first (naive) approach;

Generate x, y, z as random numbers from [-1, 1].
Normalize vector: divide x, y and z by the vector length to make it unit vector.

Clojure implementation:

(defn rand-3d []
  (let [x (dec (rand 2))
        y (dec (rand 2))
        z (dec (rand 2))
        len (Math/sqrt (+ (* x x) (* y y) (* z z)))]
    [(/ x len) (/ y len) (/ z len)]))

Visulization:

Unfortunately my visualization sucks. I couldn't come up with a better technique to draw vectors in 3D. Red/green/blue lines are X/Y/Z axis. It's kinda hard to see if the vectors uniformly distributed or not. Let's try other approaches. I'll skip code for the second approach (it's equivalent to 2D), here is the visualization:

As for the third approach, in 3D we have to use spherical coordinates. In spherical coordinates vector is defined by three numbers (r, φ, θ):

The algorithm is the following:

Set r = 1. Generate θ as random number from [-π/2, π/2], φ from [0, 2π].
Calculate x = r*cos(θ)*cos(φ), y = r*cos(θ)*sin(φ), z = r*sin(θ).

Clojure implementation:

(defn rand-3d-v3 []
  (let [phi (rand (* 2 Math/PI))
        theta (- (rand Math/PI) (/ Math/PI 2))]
    [(* (Math/cos theta) (Math/cos phi))
     (* (Math/cos theta) (Math/sin phi))
     (Math/sin theta)]))

Visualization:

Results surprised me. My mind tells me that the third approach is the best, but my eyes prefer the first approach. The visualization from the first approach feels more uniform, I see fewer clusters. But maybe it's inefficiencies of my 3D visualization: some vectors are close to the camera and they look bigger and more clustered while the ones which further look smaller. If you have ideas how to fix it - please leave comments. I'm curious how better I can visualize it.

Conclusion

If you need to generate random vectors think carefully if your approach produces uniformly distributed vectors (if it matters for your application). I like the approach with polar/spherical coordinates because I can "see" it. If you google around you'll find suggestions to use gaussian distribution but to me it's harder to understand.

Also while working on article I found embarrassing things:

random-2d and random-3d were supposed to be released in Quil 2.6.0 but they're not there. Apparently I built the release from earlier commit.
Quil random-3d has bug and it produces only vectors with positive z coordinate:

If you want to play with generation and animation you can use Quil sketch editor for Clojurescript: 2D sketch and 3D sketch. If you prefer Clojure - checkout this project.

Getting Started with Self-Hosted ClojureScript. Part 3.

Fri, 11 Mar 2016 00:00:00 -0800

Part 1
Part 2
Part 3

In previous parts of this tutorial we learned how to compile ClojureScript code using only ClojureScript. In this article we'll see how we can optimize this process by using compilation cache.

Cache in self-hosted cljs

If you read documentation for *load-fn* carefully you probably noticed that one of the values we can pass to to callback function is :cache . When is it useful? A couple of use case come in mind:

In jsfiddle-like service. Your service might provide a set of cljs libraries to use. It makes sense to precompile all these libraries as they don't change from user to user. That's what I do for Quil: I precompile quil.core and load cache only, no source code. Fully compiling Quil using self-hosted js takes around 5 sec on my laptop which is quite a lot especially if you want to compile simple sketch.
Mike Fikes uses caching capabilities to precompile macros for Planck, OS X ClojureScript REPL.
Going further I can imagine web IDE for ClojureScript. In such IDE project might consist of dozens of cljs files and compiling them might be long. To optimize cache of each source file compilation might be stored in browser local storage and reused.

Let's cache now! The first question is where do we get cache from? Obviously, we need to compile source first and somehow get cache from that compilation. Here is the way I did it: you can pass :cache-source function in compilation opts map. This function will be called once each namespace is compiled and compiler passes a map that contains compiled source and analysis cache. For now, let's simply log the map in console:

(def deps {
  'my.math "(ns my.math) (defmacro triple [x] (* 3 x))"})

(def source "
  (ns my.test (:require-macros my.math))
  (def foo (my.math/triple 5))")

(defn load-inlined [opts cb]
  (println "Loading dependency" opts)
  (if-let [dep-source (deps (:name opts))]
    (cb {:lang :clj :source dep-source})
    (throw (js/Error. (str "Unknown namespace " opts)))))

(defn print-cache [opts cb]
  (pprint opts)
  (cb {:value nil}))

(compile source {:eval cjs/js-eval
                 :load load-inlined
                 :cache-source print-cache})

Note that print-cache takes 2 arguments: map with cache data and callback to call once we're done with processing cache. In our case we print cache and call cb immediately after that. Here is the cache we get in console:

{:lang :clj,
 :name my.math,
 :path "my/math",
 :source
 "goog.provide(\"my.math$macros\");\nmy.math$macros.triple = (function my$math$macros$triple(_AMPERSAND_form,_AMPERSAND_env,x){\nreturn ((3) * x);\n});\n\nmy.math$macros.triple.cljs$lang$macro = true;\n",
 :cache
 {:use-macros nil,
  :excludes #{},
  :name my.math$macros,
  :imports nil,
  :requires nil,
  :uses nil,
  :defs
  {triple
   {:protocol-inline nil,
    :meta
    {:file my.math,
     :line 1,
     :column 24,
     :end-line 1,
     :end-column 30,
     :macro true,
     :arglists '([x])},
    :name my.math$macros/triple,
    :variadic false,
    :file nil,
    :end-column 30,
    :method-params ([&form &env x]),
    :protocol-impl nil,
    :arglists-meta (nil nil),
    :column 14,
    :line 1,
    :macro true,
    :end-line 1,
    :max-fixed-arity 3,
    :fn-var true,
    :arglists '([x])}},
  :require-macros nil,
  :doc nil}}

Now is the time to compile using that cache. To achieve that we're going to replace load-inlined function with load-cached. load-cached is going to return cache data you saw above (I simply copy-pasted it):

(def cache
  {'my.math '{:source "goog.provide(\"my.math$macros\");\nmy.math$macros.triple = (function my$math$macros$triple(_AMPERSAND_form,_AMPERSAND_env,x){\nreturn ((3) * x);\n});\n\nmy.math$macros.triple.cljs$lang$macro = true;\n",
              :cache {:use-macros nil, :excludes #{}, :name my.math$macros, :imports nil, :requires nil, :uses nil, :defs {triple {:protocol-inline nil, :meta {:file my.math, :line 1, :column 24, :end-line 1, :end-column 30, :macro true, :arglists (quote ([x]))}, :name my.math$macros/triple, :variadic false, :file nil, :end-column 30, :method-params ([&form &env x]), :protocol-impl nil, :arglists-meta (nil nil), :column 14, :line 1, :macro true, :end-line 1, :max-fixed-arity 3, :fn-var true, :arglists (quote ([x]))}}, :require-macros nil, :doc nil}}})

(def source "
  (ns my.test (:require-macros my.math))
  (my.math/triple 5)")

(defn load-cached [opts cb]
  (println "Loading cached dependency" opts)
  (if-let [dep (cache (:name opts))]
    (cb {:lang :js :source (:source dep) :cache (:cache dep)})
    (throw (js/Error. (str "Unknown namespace " opts)))))

(defn print-cache [opts cb]
  (pr opts)
  (cb {:value nil}))

(compile source {:eval cjs/js-eval
                 :load load-cached
                 :cache-source print-cache})

Console output:

goog.provide('my.test');
goog.require('cljs.core');
my.test.foo = (15);

It works! Few notes:

We never pass the source of my.math to the compiler. Instead we pass compiled js and analysis cache.
When using cache we provide compiled js so :lang must be set to :js, not :clj as we did before.
If experimenting with it yourself - don't forget to add quote ' before cache map because it contains symbols and without quoting ClojureScript is going to try and resolve them thinking that they're actual variables.
A map passed to :cache-source doesn't tell you if a namespace is macro or regular. Check the cache above and you'll see that there is no :macro true or similar in it. It might be problematic if you have some namespace compiled both as macros and as regular namespace. For example in Part 2 we had my.math both as macro and regular namespace. In that case :cache-source will be called twice and you'll have to figure out if given cache is for macro or regular namespace. One workaround is to check (:name (:cache opts)) which has $macros at the end if it's a macro namespace.

In your real application you probably want to gather caches of all namespaces and save them somewhere for later use. There are various ways to do it. For example, I collect all caches into an atom and once everything compiled I save the content of the atom in a file on disk (I'm running on nodejs so I have access to filesystem).

Conclusion

I hope this tutorial was useful and helped to start working with self-hosted cljs. Personally, I believe it is important to have a way to experiment with self-hosted cljs without distractions of loading files from a filesystem, understanding the difference between clj, cljs, cljc files and other. The compiler doesn't care about it at all. That's what I tried to do in this tutorial: having single file that contains all necessary data to play with self-hosted cljs. If you're interested in more advanced/practical things here are some tips:

Checkout amazing Mike Fikes' blog: http://blog.fikesfarm.com/. It contains a lot of insights on various self-hosted cljs things.
Checkout articles on ClojureScript wiki about bootstrapped cljs.
Checkout replumb library that does some plumbing of self-hosted cljs infrastructure.
If you have problems, ask on ClojureScript mailing list. There are many experienced people (e.g Mike) who're willing to help.
If something doesn't work as you expect - it might be a bug. There are still some rough edges in self-hosted cljs and you might stumble upon one. Try to prepare minimal reproducible example and send it to the mailing list.
Try reading and debugging js.cljs source code. It's not easy but very useful if you get some issues with cljs compilation.

Have fun with self-hosted cljs!

Getting Started with Self-Hosted ClojureScript. Part 2.

Mon, 07 Mar 2016 00:00:00 -0800

Part 1
Part 2
Part 3

In Part 1 we learned how to compile simple cljs namespaces. Here is what we had at the end of the previous article:

(ns my.main
  (:require [cljs.js :as cjs]
            [cljs.pprint :refer [pprint]]))

(enable-console-print!)

(def state (cjs/empty-state))

(defn compile [source opts]
  (cjs/compile-str state source "bla" opts #(println (:value %))))

(def source "
  (ns my.test)
  (defn triple [x] (* x 3))
  (triple 5)")

(compile source {:eval cjs/js-eval})

Let's modify callback function in compile function so it prints compiled code if it was successful or error otherwise:

(defn print-result [res]
  (if (:value res)
    (println (:value res))
    (pprint res)))

(defn compile [source opts]
  (cjs/compile-str state source "bla" opts print-result))

Adding some :require

Now we're going to use functions from other namespaces. We're going to move function triple into separate namespace my.math and require my.math from my.test. When cljs compiles namespace and sees that it requires another namespace - the compiler needs to get the source of that dependancy namespace as well. The compiler doesn't know how to get them (all it knows is how to compile). So we have to provide a load function that given namespace name loads source of the namespace. Usually, this load function fetches clj/cljs/cljc files from disk. We want to keep it simple so we're going to inline source of my.math in our file without loading it from anywhere. Here is the code:

(def deps {
  'my.math "(ns my.math) (defn triple [x] (* 3 x))"})

(def source "
  (ns my.test (:require my.math))
  (my.math/triple 5)")

(defn load-inlined [opts cb]
  (println "Loading dependency" opts)
  (if-let [dep-source (deps (:name opts))]
    (cb {:lang :clj :source dep-source})
    (throw (js/Error. (str "Unknown namespace " opts)))))

(compile source {:eval cjs/js-eval :load load-inlined})

Console output:

Loading dependency {:name my.math, :path my/math}

goog.provide('my.test');
goog.require('cljs.core');
goog.require('my.math');
my.math.triple.call(null,(5));

Note that final output contains only compiled source, it doesn't include compiled my.math which is a dependency. Compiler didn't actually compile my.math, just analyzed it. If you try adding a typo, e.g. change (my.math/triple 5) to (my.math/truple 5) you'll get an warning in console:

Loading dependency {:name my.math, :path my/math}
WARNING: Use of undeclared Var my.math/truple at line 3

goog.provide('my.test');
goog.require('cljs.core');
goog.require('my.math');
my.math.truple.call(null,(5));

You see that source is still compiled with truple but the compiler warned us that it haven't found truple function in my.math namespace. Note that content of my.math namespace doesn't affect compilation of source in any way. We could even return an empty string for my.math. In that case, source would still be compiled the same way, but the compiler would warn us that we're using undeclared functions.

You can add more dependencies. For example my.math might require additional namespaces and compiler will try to load them using the load function we provided. Even if you require some standard namespaces like clojure.string - compiler still asks us to provide the source. We can either load real source file from ClojureScript jar file or just return an empty string and ignore compiler warnings about calling undeclared functions clojure.string.

How about macros?

One really cool feature of self-hosted cljs is that it can compile and expand macros. Macros are an important part of Clojure/ClojureScript ecosystem and without this feature, self-hosted cljs can't be truly self-hosted. Macros in self-hosted cljs work similar to regular ClojureScript: source is a regular cljs file that contains definitions of functions, variables, calls functions from other namespaces and also may call macros. But it cannot define macros: if you add defmacro to cljs source it will be ignored and you won't be able to use it. Macros must be defined in separate "macros" namespaces. With regular ClojureScript you put macros in .clj files not .cljs. In self-hosted cljs it is similar but instead of .clj vs .cljs files there is macros vs regular namespaces. How does compiler load macros namespaces? Using the load function we passed in compile options map. For macros namespaces, it adds :macros true so we (implementors of load function) can understand which type of namespaces compiler needs. Let's see it in practice.

We're going to turn triple function into a macro: it's going to perform multiplication during compile time instead of runtime. This macro is going to work only if you pass a number. Code hasn't changed much:

(def deps {
  'my.math "(ns my.math) (defmacro triple [x] (* 3 x))"})

(def source "
  (ns my.test (:require-macros my.math))
  (def foo (my.math/triple 5))")

(defn load-inlined [opts cb]
  (println "Loading dependency" opts)
  (if-let [dep-source (deps (:name opts))]
    (cb {:lang :clj :source dep-source})
    (throw (js/Error. (str "Unknown namespace " opts)))))

(compile source {:eval cjs/js-eval
                 :load load-inlined})

Note that we now use :require-macros instead of :require in my.test and triple is now defined as defmacro, not defn. Console output:

Loading dependency {:name my.math, :macros true, :path my/math}

goog.provide('my.test');
goog.require('cljs.core');
my.test.foo = (15);

Compiler applied triple macro to number 5 during compilation and we got 15 even without evaluating compiled code! Also take closer look at Loading dependency {:name my.math, :macros true, :path my/math}. It now has :macros true which wasn't there before. This means that compiler asked us to load macros my.math namespace, not a regular namespace. It is possible that same namespace is loaded "twice": as macros and as regular namespace. But it doesn't mean that we have to return the same source for both cases. On the contrary, usually "macros" namespace has, well, some defmacro in it and regular namespaces has regular cljs functions. We'll see such example in a little bit.

How does cljs apply macros? Roughly the process is following: the compiler compiles macros namespace to js, defmacro becomes js functions with additional properties, then it uses these freshly evaluated js function whenever it finds macro calls. So far we've seen only final compiled js from source, but under the hood the compiler also compiles my.math namespace because now it's macro namespace. Is there a way to see it so we can better understand how it works? Fortunately, there is. You can add :verbose true to compile options and it will enable internal debug logs. It is super useful if you're trying to debug some issue with compilation in dependencies. Let's see what we get:

(compile source {:eval cjs/js-eval
                 :load load-inlined
                 :verbose true})

Console output:

Namespace side effects for my.test
Processing :use-macros for my.test
Processing :require-macros for my.test
Loading my.math macros namespace
Loading dependency {:name my.math, :macros true, :path my/math}
Evaluating my.math
Namespace side effects for my.math
Processing :use-macros for my.math
Processing :require-macros for my.math

goog.provide("my.math$macros");
my.math$macros.triple = (function my$math$macros$triple(_AMPERSAND_form,_AMPERSAND_env,x){
return ((3) * x);
});
my.math$macros.triple.cljs$lang$macro = true;

goog.provide('my.test');
goog.require('cljs.core');
my.test.foo = (15);

It might be somewhat noisy but take a look at the last 2 chunks of code. The first defines my.math$macros namespace which is our macros namespace that compiler used under the hood. You can see that it has special $macros suffix that helps to keep it separate from regular namespaces. The last chunk of code is compiled source which we already saw, so nothing interesting here. With :verbose output compiler prints all code it compiles and evaluates which is cool.

Let's try now load the same namespace twice: as macro and as regular namespace. We're going to fix triple macro so it does following: if the argument is a number - perform multiplication during compile time, otherwise (if it is a variable or call to other function) defer calculation to triple-fn function. In order to do that, we need to fix our load-inlined function. Right now it is dumb and cannot distinguish between macros and regular namespaces, but in real life you should distinguish between these two cases. Actually, documentation for *load-fn* suggests that when loading macros namespace you should use only .clj or .cljc files and for regular namespaces .cljs or .cljc files. In our case, we're going to have 2 maps of inlined namespaces: deps-macros and deps-regular. Here it is:

(def deps-macros
  {'my.math "
  (ns my.math)
  (defmacro triple [x]
    (if (number? x)
      (* 3 x)
      `(my.math/triple-fn ~x)))"})

(def deps-regular
  {'my.math "(ns my.math) (defn triple-fn [x] (* 3 x))"})

(def source "
  (ns my.test (:require my.math) (:require-macros my.math))
  (def five 5)
  (+ (my.math/triple 5) (my.math/triple five))")


(defn load-inlined [opts cb]
  (println "Loading dependency" opts)
  (let [deps (if (:macros opts) deps-macros deps-regular)
        source (deps (:name opts))]
    (if source
      (cb {:lang :clj :source source})
      (throw (js/Error. (str "Unknown namespace " opts))))))

(compile source {:eval cjs/js-eval
                 :load load-inlined
                 :verbose true})

Console output (debug info omitted):

Loading dependency {:name my.math, :macros nil, :path my/math}
Loading dependency {:name my.math, :macros true, :path my/math}

goog.provide("my.test");
my.test.five = (5);
((15) + my.math.triple_fn.call(null,my.test.five));

It worked! You can see that my.math was loaded twice. First as regular namespace and second time as macro namespaces. From compiled code we see that the first call to triple was calculated in-place while the second was deferred to triple-fn. Just like we expected. But to be honest, this example is not complete because even though my.math was loaded twice, the compiler compiled only macro namespace and didn't compile regular namespace, just analyzed it. To make it fully work replace cjs/compile-str with cjs/eval-str in compile function and now it should fully compile and evaluate each namespace and produce "30" in console output. Also, with verbose output you should see all 3 namespaces being compiled under the hood.

Initially, I wanted to make this example more elegant by making triple both macro and function. This way we can use triple anywhere. Where possible compiler precomputes and where it is not possible - defers it to runtime triple function. I tried following:

(defmacro triple [x]
  (if (number? x)
    (* 3 x)
    `(my.math/triple ~x)))

But the problem here that when compiler processes (my.math/triple five) - it thinks that it is macro and expands it to (my.math/triple five) and ... expands it again. And again. And again. And many more times until we're out of stack. Basically, the compiler doesn't know if we use my.math/triple as macro or function. One way hack I found is to make "function" call look more like js call and not regular clojure:

(defmacro triple [x]
  (if (number? x)
    (* 3 x)
    `(.triple js/my.math ~x)))

This way the compiler doesn't try to expand result as macro. But it doesn't look clojurish and I decided to go with more clunky triple-fn instead. I don't know if there is a better way to do it.

That's all for today. We learned how to compile both regular cljs and macros in self-hosted cljs! In the next article, I'll show how to optimize compilation using cache.

Getting Started with Self-Hosted ClojureScript. Part 1.

Sat, 05 Mar 2016 00:00:00 -0800

Part 1
Part 2
Part 3

ClojureScript is a compiler for Clojure that targets JavaScript. About 7 months ago it got a nice feature that allows you to compile ClojureScript code using ClojureScript: self-hosting. That means that we can now compile ClojureScript fully in browser! That's pretty cool for creating various interactive in-browser tutorials/repls/workspaces/whatever. I've been working on compiling Quil using self-hosted cljs (including macros) and decided to write basic tutorial of how to work with self-hosted cljs. This tutorial uses vanilla cljs without any additional libraries or features like reading files from disk, sending XHR so all you need is to understand basic ClojureScript syntax. In these articles I'll be using latest ClojureScript currently available: 1.7.228.

Prepare

Let's start with creating a basic cljs project.

git clone https://github.com/nbeloglazov/blog-projects.git
cd blog-projects/self-hosted-cljs
lein cljsbuild auto

Open index.html in your browser. Alternatively you can create or reuse any existing cljs project your have.

Your first compile

Let's start with standard "Hello, world". If you cloned my project the code is already there:

(ns my.main
  (:require [cljs.js :as cjs]
            [cljs.pprint :refer [pprint]]))

(enable-console-print!)

(def state (cjs/empty-state))

(cjs/eval-str state
              "(.log js/console \"Hello, world\")"
              "bla"
              {:eval cjs/js-eval}
              identity)

Reload the page and you should see "Hello, world" in console. So let's see what's going on here:

Require cljs.js namespace. That namespace contains all self-hosting-related functions.
Define state variable that is initialized with empty state. Tbh I don't know what state can contain so I usually pass empty state to all eval/compile functions.
Calling eval-str to eval cljs code. It takes quite a few of arguments. But the important ones for us here is second (source) and fourth (opts) arguments. Other are "default" and we can ignore them for now. If curious - check docs to see what each argument is. In options map we provide a function that is used to evaluated compiled js. Cljs compiler doesn't have default eval function so we have to provide it explicitly all the time. If you take a look at js-eval function you'll see that it is plain old javascript eval() (as name implies), no magic here.

Evaluation is good, but it would be more interesting to see compiled code before it is evaluated. It helps to understand how it works and makes it easier to debug. To see compiled code of eval-str we'll be using compile-str:

(cjs/compile-str state
                 "(.log js/console \"Hello, world\")"
                 "bla"
                 {:eval cjs/js-eval}
                 #(println (:value %)))

And output in logs:

console.log("Hello, world");

The last (fifth) argument is changed from identity to #(println (:value %)). This argument is a callback function that will be invoked once cljs finished compiling. The result will be passed to the provided callback. In the first example with eval-str we didn't really care about the result, we just wanted console.log to be evaluated. But now we use compile-str and we want to see result of compilation, so we pass a function that takes that result and prints it to console.

Now let's compile real namespace with multiple functions. Also I'm going to introduce helper compile function to omit default arguments:

(defn compile [source opts]
  (cjs/compile-str state source "bla" opts #(println (:value %))))

(def source "
  (ns my.test)
  (defn triple [x] (* x 3))
  (triple 5)")

(compile source {:eval cjs/js-eval})

And result is following:

goog.provide('my.test');
goog.require('cljs.core');
my.test.triple = (function my$test$triple(x){
return (x * (3));
});
my.test.triple.call(null,(5));

Compiled code readable and you can pretty easily relate cljs source with compiled js. It has some goog.provide and goog.require magic which comes from Google Closure compiler used by ClojureScript, but it is quite simple: compiled code declares namespace my.test and uses namespace cljs.core. Everything else is standard javascript.

Thats all for now. Now we can compile basic cljs namespaces. In the next article we'll be compiling code consisting from multiple namespaces/files and using macros.

Live Reloading in Quil Cljs

Mon, 15 Jun 2015 00:00:00 -0700

Quil supported live reloading since forever and it has been one of its coolest features. But only in Clojure. ClojureScript version lacked the support until now. Latest Quil release (2.2.6) fixes the problem. Now if you use plugins for live coding in ClojureScript such as figwheel - you'll be able to reload Quil functions (draw, update, key handlers) on fly without reloading the browser. Here is 3 steps instructions how to do it:

1 Create new Quil cljs project: lein new quil-cljs hello and go to hello folder.
2 Setup figwheel in project.clj:

  ...
  :plugins [[lein-cljsbuild "1.0.6"]
              [lein-figwheel "0.3.3"]]
  ...
  :cljsbuild {:builds [{:source-paths ["src"]
                          :figwheel true
                          :compiler {...}}]}

3 Start figwheel: lein figwheel.

That's all. Now open index.html and you should see a ball rotating around the center of the sketch. When you update core.clj and save it - the updates will be reflected in the browser. For example change rotation direction by modifying update function: replace plus with minus: (- (:angle state) 0.1) and save it. The ball will start rotating in counter-clockwise direction.

Auto-reloading is supported only in defsketch. If you use sketch function - it won't work (the same as in Clojure version). When defsketch macro is used, all functions (draw, update, etc) are wrapped into anonymous functions so they're called by name rather than by value. This allows for reloading to work: when hello.core/draw is reloaded by figwheel, quil will call the anonymous function wrapper which in turn calls hello.core/draw. It adds some overhead so use sketch instead of defsketch in "production".

Trees, Quil and Random

Mon, 18 May 2015 00:00:00 -0700

Last week I've been playing with random tree generation in Quil. Idea is simple: generate a tree that resembles a real tree and doesn't make you think that it was generated by computer. Suprisingly it's not that hard to do. Tree is represented as a collection of branches. A branch consists of 3 components: start point, angle (from 0 to 2π) and length. Each branch can have 0, 1 or 2 sub-branches. Algorithm for generating sub-branches from a branch:

Each branch can have up to 2 sub-branches. Probability of each sub-branch is 90%.
For each sub-ranch start point is calculated as end point of parent branch.
Sub-branch lengths are randomly selected from [0.7L, 1.0L], where L is parent's length.
Sub-branch angles are randomly selected from [α-¼π, α+¼π], where α is parent's angle.
Add sub-branches to the tree.

Now, having the function for generating sub-branches, tree generation is easy:

Create root: the first branch with angle ½π.
Branch-off each branch on previous level.
Repeat step 2 as many times as needed. My trees have 14 levels.
Concat all levels into single collection and draw them.

Last part is especially easy to do in clojure and each step corresponds to one line:

(defn generate-tree [root levels]
  (->> [root]
       (iterate #(mapcat branch-off %))
       (take levels)
       (apply concat)))

That's it. Now more trees!

White trees on black background look especially cool:

Quil tips

I've shown only static images of trees, but in reality they're animated so you can see how tree grows from a single root to a full tree with thousands of branches.

Initially I used a simple approach and was redrawing all branches on each frame. It works well when you have small number of branches but tree with 14 levels has 7000-10000 branches and drawing 10000 elements on each frame is well... slow. FPS drops from 60 to 10. So I optimized it a little, removed clearing background and changed draw-branch function to skip branches that are already fully drawn. That helped drastically bringing FPS back to 60. The big drawback that you can't do nice transformations now that may require redrawing everything. For example after tree is drawn I wanted to zoom into smallest branch and start drawing new tree from that branch, but it requires redrawing everything during zoom and I gave up this idea.

Another trick I did is around fading background color into tree color (see animations below). The idea is that screen background gradually turns to the same color as tree essentially clearing the tree. To achieve this the tree is drawn on a separate graphics object. Graphics object is transparent and when you copy it to main screen you can see background. With this technique draw function looks like the following:

(defn draw [state]
  ; Clear screen using calculated background screen. Background depends
  ; on tree state: if tree is fully drawn it gradually changes to match the
  ; tree color.
  (q/background (calculate-background-color state))

  ; Draw active branches on graphics object. Note that we don't erase
  ; existing branches, draw-tree only draws active branches. 
  (q/with-graphics (:buffer state)
    (draw-tree state))

  ; Copy buffer graphics to main screen.
  (q/image (:buffer state) 0 0))

And finally endless trees:

Controls (tree must have focus):

r - regenerate tree;
up/down - change number of levels;
i - toggle stats;
s - save as image (opens in new tab);

You can find source code on GitHub or try it online on quil.info.

Mongo Backup Using Clojure

Mon, 15 Dec 2014 00:00:00 -0800

In this post I'll show a simple tool that creates Mongo backup and uploads it to Google Cloud Storage. Suppose you have a small webproject that uses mongo. The project runs on a single machine on some cloud provider. And you want to have periodic mongo backups. Potentially you can achieve it using features of your provider like disk snapshots, but it may require shutting down the machine or unmounting disk and it feels heavy and not fun. Instead we'll create our own tool that periodically creates backups and uploads them to a file hosting service like Dropbox or Google Drive or (in my case) Google Cloud Storage. I use Google Cloud Storage because I'm already using Google Cloud Platform as hosting for Hatnik so why not to use available services?

The tool is relatively small and consists of 2 parts: backup creation and uploading.

Create backup

Assumptions: mongo instance doesn't require authentication and OS is linux.

(require '[clojure.java
            [shell :refer [sh with-sh-dir]]
            [io :refer [file]]])

(defn clean
  "Removes old dump from current folder if exists."
  []
  (sh "rm" "-r" "dump")
  (sh "rm" "dump.zip"))

(defn create-dump
  "Creates dump and pack it into dump.zip file in current folder."
  []
  (clean)
  (sh "mongodump")
  (sh "zip" "-r" "dump.zip" "dump"))

(defn archive-name
  "Generates name for dump file. That's how it will be saved in Cloud Storage."
  []
  (-> (java.text.SimpleDateFormat. "yyyy-MM-dd_kk-mm")
      (.format (java.util.Date.))
      (str ".zip")))

As you can see it is pretty trivial. We use mongodump utility for creating dump from running mongo instance. Mongo docs say that it should be used only for small deployments as it hurts performance.

Uploading to cloud

Now we're going to jump through Google API hoops in order to upload single zip file. First of all you should setup service account that will allow the tool to upload to the storage. It will generate an email address and private key that will be used for authentication. Now let's go to code. To simplify things we will be using official java library for working with Google Cloud Storage (javadocs). In the code I omit all imports as there are quite a few of them and you can get them from github project at the end of the post.

(def email-address "<YOUR SERVICE ACCOUNT EMAIL HERE>")
(def p12-file (file "<PATH TO PRIVATE KEY FILE>"))
(def bucket-name "mongo-backups")
(def app-name "<YOUR GOOGLE CLOUD PROJECT NAME>") ; not sure if it's actually needed

(defn authorize
  "Authorizes app and returns credentials object that will be used to
  create storage object."
  []
  (.. (GoogleCredential$Builder.)
      (setTransport (GoogleNetHttpTransport/newTrustedTransport))
      (setJsonFactory (JacksonFactory/getDefaultInstance))
      (setServiceAccountId email-address)
      (setServiceAccountScopes [StorageScopes/DEVSTORAGE_FULL_CONTROL
                                StorageScopes/DEVSTORAGE_READ_ONLY
                                StorageScopes/DEVSTORAGE_READ_WRITE])
      (setServiceAccountPrivateKeyFromP12File p12-file)
      (build)))

(defn get-storage
  "Authorizes and creates storage object."
  []
  (.. (Storage$Builder. http-transport json-factory (authorize))
      (setApplicationName app-name)
      (build)))

(defn upload-zip-file [name zip-file]
  (let [client (get-storage)
        object (doto (StorageObject.)
                 (.setName name)
                 (.setContentType "application/zip"))
        content (FileContent. "application/zip" zip-file)]
     (.. client
         (objects)
         (insert bucket-name object content)
         (execute))))

And finally glue both parts together:

(defn create-and-upload-dump []
  (with-sh-dir "/tmp"
    (create-dump)
    (upload-zip-file (archive-name) (file "/tmp/dump.zip"))
    (clean)))

; main method needed to run the tool from command line
(defn -main [& args]
  (create-and-upload-dump))

That's all. You can test the tool by executing create-and-upload-dump function from repl.

Create cron job

Now we can generate executable jar using lein uberjar, copy it to some directory on the server together with private key file and create daily cron job by creating script in /etc/cron.daily folder. My cron job looks like this:

#!/bin/sh
#
# Daily mongo backups

WORKING_DIR=/home/nbeloglazov/backup
cd $WORKING_DIR
java -jar $WORKING_DIR/backup.jar

I suspect that if Dropbox or something else is used instead of Google Cloud Storage then code should be simpler. Authentication in Google APIs is kinda messy and it's not easy to find correct combination and order of methods to call to make it work.

For reference here is a complete working version of the tool: project on GitHub.

GitHub Actions in Hatnik

Mon, 24 Nov 2014 00:00:00 -0800

Hatnik is a tool that allows you to track library releases: you want to know when new version of ClojureScript released - you go to Hatnik, setup an action and next time ClojureScript released the action will be performed. Initially only email action was supported. Happy to announce that now we support GitHub Issue and Pull Request actions.

Issue Action

This action is pretty trivial: create an issue in selected repo when library is released. You can customize repo (obviously), issue title and body. Title and body support template variables: for example "{{library}} {{version}} released" will be rendered as "org.clojure/clojurescript 0.0-2371 released".

Issue action might be useful, for example, for library maintainers. Usually there are quite a few tasks you need to do when you release new version: update wiki, tutorials, some post-release testing. It is quite easy to forget something so Issue action can help with that: for each task create an issue and then later you or someone else can pick it up and do whatever needed to be done.

Pull Request Action

Pull Request action modifies public repo and opens pull request. Each modification operation is a find-and-replace operation and consists of 3 fields: file to be modified, regex and replacement. Regex and replacement fields support template variables.

Pull request can be used to keep dependencies of you project up-to-date: just create an action that updates version of dependency X in project.clj each time X is released. It is especially cool when you have CI (e.g. Travis) that runs tests on pull requests: you can immediately see if new version breaks your application and if it doesn't break - simply merge it. To keep your application up-to-date you need just one click! Another use case is for library maintainers: usually library README includes latest version of the library so users can simply copy-paste it to project.clj and start using it. But these versions sometime become out-of-date as maintainers forget to update README. Pull Request action can help with that. For example I set up a few actions for Quil that update versions in README, examples and lein templates. Here is an example of such pull request. Final note about pull requests - if pull request action doesn't change anything in repo (for example file moved or version in README has been already updated) - you'll get an email that describes why Hatnik couldn't open a pull request.

TODO

Import build files. Currently Hatnik doesn't provide options to import build files like project.clj. So if a project has a lot of dependencies it becomes tiring to create actions for each dependency one by one. Hatnik should support import capabilites.
Allow to use temporary accounts to create restricted set of actions and decide whether to use Hatnik or not. Currently you have to login via GitHub in order to try it out.
Support more languages. Currently only maven-compatible libraries are supported (Java, Clojure, Scala, Groovy). We should support more.

Contributing

Hatnik is an open source project. Take a look at the repo. Bug reports and pull requests are very welcome!

Introducing Hatnik

Wed, 01 Oct 2014 00:00:00 -0700

Yet another blog post about a ClojureCup 2014 project. But this time from me. About Hatnik!

Don't miss a release

Hatnik is a web app we created during ClojureCup weekend. It can notify you about new library releases (currently via email only). It supports Clojure/Maven libraries which are hosted on Clojars or Maven Central. How to use it: sign in, create an action in which you specify which library you want to watch and template for the email and that's all. Next the library is released we'll send you an email. Try it out: hatnik.clojurecup.com. If you don't like signing in unfamiliar websites which you might never visit again, here is a screenshots-demonstration:

Initial page after you logged in

Create new action

Action created

Libraries used by Hatnik

You can also group actions to projects. Projects is just a named group of actions. Each user by default has "Default" project (see screenshots) so you might ignore it.

Future plans

We did the project over 2 days so obviously there are a lot of things to improve.

More actions

Currently Hatnik supports only email notifications. We're planning to add more. Here is the initial list:

Noop - does nothing. Might be useful for creating dashboard-like project that simply displays latest versions.
GitHub Issue - creates an issue on GitHub repo.
GitHub Wiki Page - changes GitHub wiki page using simple find-and-replace mechanism.
GitHub Pull Request - creates pull request that modifies some files using find-and-replace mechanism.

Cleanup

We developed Hatnik in hurry so there is some technical debt.

Improve web client. We used Om for the web client but we didn't know it good enough so in some places code is not Om-like at all. For example we spend few hours at the end of the contest trying to fix a bug, which made library textfield read-only. Looks like the bug is described in tutorial, but even using the tutorial we couldn't make it work. So we ended up with horrible hack that sets value to textfield using setTimeout.
Add more validation. Currently REST API is not strict, it does very little validation. It should do more. We're looking at Prismatic schema for that.
Add core.typed. At work I'm using Google Closure Compiler quite heaviliy. Among other features it introduces type system to JavaScript. It makes much easier to write correct code. So I'd like to try core.typed for same reasons and see whether it helps or not me.
Add tests. Currently we have some DB tests. We should add integration-like tests for API and ideally sending emails, creating github issues. Tests for client side would be nice, may be something WebDriver-based.

Other plans

Hatnik currently supports only maven-compatible libraries. Nothing stops it from supporting libraries from other languages/platforms, like Ruby, JavaScript, Python. I think most tools in other languages also use HTTP for downloading libraries from central repository, so it should be pretty easy to implement in Clojure.

Currently all actions have to be created manually one by one. It would be cool to be able to provide a link to project.clj and it will be parsed automatically. Again, nothing prevents us from supporting other projects files like pom.xml, build.sbt, Gemfile. Some of them just harder to parse as they're written in DSL, but still, should be fun task to do.

Afterwords

Few words about the project name. Hatnik is a house spirit in Slavic folklore. There are different versions of what he does and whether he good or not, but in my environment Hatnik is a good grandpa-looking-like spirit which helps with household. So we wanted the app to do the same - help with some simple tasks which help to keep project dependencies or documentation in order. I hope it will be useful for you. If you liked Hatnik you can vote for us. Also have a look at other projects, they're pretty interesting. And they also needs votes ;)

Hatnik is open-sourced and here is GitHub Repo.

Halfway Through ClojureCup

Sat, 27 Sep 2014 00:00:00 -0700

Hatnik

As you might have guessed, this year I'm participating in ClojureCup. We (me and Maksim) are working on a project called Hatnik. It's a web app which aims to help you to track library releases. The idea is pretty simple: setup actions to be performed once some library is released. An action consists of 2 parts: library to watch and an instruction to perform. Example: send email to me@nbeloglazov.com when org.clojure/clojurescript releases. It is somewhat similar to IFTTT (but more specialized) and pretty similar to Artifact Listener (but obviously cooler!). We have plans to support following actions: email, github issue, github pull request, change wiki on github. Initially we're going to support only email and add others if we have enough time during the contest.

Organizers provided quite a few additional services like private github repo, a server for hosting app, CI servers, error-tracking utility, team chat and some other stuff. We've been using only few of them: github repo, server for hosting app and Flowdock for communication: it has integration with github and trello so we can see everything that happens in repo or task list in realtime. Though other services looked intersting I felt we'd spent too much time setting them up instead of working on the project.

Architecture

The app consists of 3 parts:

Web client - that's what user sees and interacts with.
Web server - provides API for web client and stores data to DB.
Worker server - periodically checks versions of all registered libraries and performs actions.

I work on servers while Maksim works on the web client. For the web client we decided to use ClojureScript + Om and Twitter Bootstrap for fancy styling. Server side libraries are pretty standard: ring, compojure, monger and few other libs for emails, github api, and other stuff. You can find more details about the project in design doc which we prepared beforehand so we can concentrate on coding!

What we have so far

I would say we have quite enough:

Web client

Client can load and display projects together with actions. You can add new projects. What's left to do: actions adding/modifying, login (everything is ready, only need to add buttons), nice "About" page that describes how to use Hatnik.

Web server

Web server is mostly done. REST API is implemented, everything is stored in DB. DB is represented as a protocol that has 2 implementations: in-memory and mongo. It might be overkill but I found it's quite useful to be able to run everything in memory. I even had time to add tests for DB! I used Postman for testing REST API. It is quite cool as it allows you to save requests and reuse them later. I probably should have added some tests for REST instead but due to the time limit I decided to go with Postman: it is easier to setup and use.

Worker server

Currently worker server can only submit emails on user request: when you create an action you have an option to test it, the test request is sent to the worker server to perform the action. What's left to do: add logic for checking versions of all registered libraries and perform actions for updated libraries, invoke this logic periodically.

Conclusion

I'm having a lot of fun! I don't remember when I wrote so much code last time. Lately I've been coding less at work so ClojureCup came just in time. Especially when you have an excuse to not write tons of tests and just code for the sake of coding. If you're interested in what we have so far - check our github repo. And of course check out our app when the contest is over.