Jim Cheung

Thursday, June 02, 2016

read again the How to Use Emacs section from Clojure for the Brave and True and cider documentation: Using the REPL, learned a few new tricks:

emacs:

cider

docs

history

to save repl to a file after cider-quit

(setq cider-repl-history-file "path/to/file")

to assign last eval to a variable, use *1, *2 ..., *e for last exception

(def x *1)

Friday, June 03, 2016

not just me have slow cider-jack-in problem: Do some profiling to find out why it takes so long to start a REPL ยท Issue #1717

porting some java stuffs to kotlin, it's such a pleasant to work with. Emacs doesn't have any kotlin support, I just use intellij idea, not bad after I figured out how gradle works under idea. I like that idea will convert to kotlin codes when I pasted java codes into a kotlin file, it's not 100%, but basic struture is there and much better than write from scratch.

Saturday, June 04, 2016

I love ncurses apps, I'm using mc and cmus daily.

today I just did a apt-cache search ncurses, tried few more apps:

Sunday, June 05, 2016

found quite good looking font for terminal: F25 Bank Printer Font

more networking books to read, borrowed another classic: Effective TCP/IP Programming: 44 Tips to Improve Your Network Programs from local library.

Monday, June 06, 2016

java convesion between string and timestamp, without using clj-time:

(import 'java.sql.Timestamp
        'java.text.SimpleDateFormat)

(defn ts->str [ts format]
    (->> ts
         (* 1000)
         Timestamp.
         (.format (SimpleDateFormat. format))))

(defn str->ts [str format]
  (->> str
       (.parse (SimpleDateFormat. format))
       .getTime
       (* 0.001)
       long))

Tuesday, June 07, 2016

worked out something to manage dependencies of a single clj file.

when I want to start something quick or just test something on clojure, simply create a single clj feels more lightweight than lein new and cider. but I can't use external libraries in this way, tools like lein-try and lein-oneoff helped not really ideal for scripting.

I checked how to download dependencies, mvn dependency:copy-dependencies is the easiest way I found, even though it requires maven ...

I wrapped up the tool and put it on github: usrjim/oneclj

Wednesday, June 08, 2016

the latest cognicast is Clojure spec with Rich Hickey - Cognicast Episode 103, the part about changes and compatible is fascinating.

however, one thing I worried is how much performance tradeoffs clojure.spec makes? and people are benchmarking it

kotlin code can be quite concise, I changed from

fun filteredBy(region: String, fn: (resultData) -> Boolean): resultData? {
    val result = getResult(region)?.filter(fn)
    return result?.first()
}

to

fun filteredBy(region: String, fn: (resultData) -> Boolean): resultData? {
    return getResult(region)?.filter(fn)?.first()
}

and finally

fun filteredBy(region: String, fn: (resultData) -> Boolean): resultData? = getResult(region)?.filter(fn)?.first()

to get recent browsed urls from firefox:

cat ~/.mozilla/firefox/xxxxxx.default/sessionstore-backups/recovery.js | jq '.windows[].tabs[].entries[]|.title+","+.url'

sharing files within an internal network, can simply start php server inside the folder:

php -S 0.0.0.0:3000

if using clojure, with my oneclj:

;;[ring/ring-core "1.5.0-RC1"]
;;[ring/ring-jetty-adapter "1.5.0-RC1"]

(use 'ring.adapter.jetty)
(use 'ring.middleware.file)

(-> {}
    (wrap-file ".")
    (run-jetty {:port 3000}))

Friday, June 10, 2016

I always want to know what Ruby Rogues' intro music is. Today when I'm listening 3 Doors Down - Kryptonite, I laughed: this is the ruby rogues music.

the clj above was using ring 1.5.0-RC1, funny that 1.5.0 was released on that day as well

it will be nice that my oneclj could pack the clj as a jar. I think all I need to figure out is the manifest file

a simple poc version:

hello.clj

(ns hello
  (:gen-class))

(defn -main[]
  (println "fat jar!"))

compile with: 

#!/usr/bin/env bash

mkdir -p classes lib
cp /opt/clojure/clojure-1.8.0.jar lib/
java -cp .:lib/* clojure.main -e "(compile 'hello)"
cat <<EOF > manifest.txt
Main-Class: hello
Class-Path: lib/clojure-1.8.0.jar
EOF
jar cfvm hello.jar manifest.txt -C classes . lib

run with: java -jar hello.jar

but seems there're some limitation: the ns, :gen-class and -main function are required. not easy to wrap a clj file with these by scripting. let's see.

Saturday, June 11, 2016

an updated poc version of packing a clj as a jar, it's working and I'm quite happy with it.

#!/usr/bin/env bash

jars=$(find lib/ -type f | sed 's/^/  /')
clojure=/opt/clojure/clojure-1.8.0.jar

mkdir -p classes lib
cp $clojure lib/

cat <<EOF > hello.clj
(ns hello
  (:gen-class))

(defn -main[]
  (load-file "sample.clj"))
EOF

java -cp .:$clojure clojure.main -e "(compile 'hello)"

cat <<EOF > manifest.txt
Main-Class: hello
Class-Path: $jars
EOF

jar cfvm hello.jar manifest.txt -C classes . lib sample.clj

will wrap it up and commit to the oneclj repository.

Sunday, June 12, 2016

so sad the way that packing stuffs in a jar does not work.

it did work because I got my dependencies/script in the same relative location so that the program could reach to those files.

it doesn't work because:

  1. put jars in manifest Class-Path won't be loaded, unless you have custom code to load all those dependencies.
  2. unjar all dependencies and pack them into a jar won't work, because you'll need to compile all clojure dependencies (AOT).

even use maven or one-jar I wonder whether they could compile clojure libraries. so I will drop it, lein and boot are the right tools to handle this problem.

lots of readings about My Increasing Frustration With Clojure and discussions on reddit and proggit

Monday, June 13, 2016

I was thinking about the jar packing problem while taking the train. coz I remembered clojure-slim.jar is an un-compiled version of clojure.jar, which means clojure.jar is already compiled to java classes and should be able to called within the jar.

also once clojure has been bootstraped, it won't be matter whether or not other clojure libraries are compiled, they can be interpreted by the main clojure library.

I suspected it didn't work because I included two clojure.jar: clojure-1.8.0.jar (by me) and clojure-1.7.0.jar (by maven).

still have another problem, how to load the clj file inside a jar?

I was using load-file but surely it can't refer to a file inside the jar.

load-script is the correct function to use. (note, the namespace is clojure.main not clojure.core), using /@sample.clj can call sample.clj inside a jar successfully.

so decided to try again, removed clojure-1.7.0.jar, unpacked all the jar dependencies and included the classpath in the manifest, changed load-file to clojure.main/load-script.

it worked. the generated jar file ran everywhere.

the fat-jar bash script was committed to usrjim/oneclj

when googling how to run clj scirpt, accidentally found this page and it has useful information about Enhancing Clojure REPL with rlwrap

this stackoverflow post has information for cleaning the repl?

to set multiple lines variable in a Makefile, can do with:

define mline =
line one
line two
some $(VAR)
endef

calling with $(mline) will treat each line as a command, a trick I found is:

export mline
@echo "$$mline"

Tuesday, June 14, 2016

added the rlwrap enhancement to oneclj, will generate completion list for auto-completion.

however, it requires use the namespace first coz now simply take functions from (all-ns). I wanted to include all functions in all dependencies, but seems another difficult task, maybe enhance later.

one project I wanted to do is analyze my photos and tag them for search. it's a deep topic, of course I will use 3rd party library of API.

google, ibm, microsoft are all doing it, can find more information in this hacker news post.

but I don't want to send my photos to them for privacy reason, I'm looking for a open-source solution. TensorFlow looks big enough and may worth spending some time with. and their image recognition tutorial does exactly what I need.

maybe need to read the book first?: Neural networks and deep learning

Wednesday, June 15, 2016

try tensorflow later until there is java support (currently python and c++)

for a comparison of Deeplearning4j vs. Torch vs. Theano vs. Caffe vs. TensorFlow, just pick one of them to try.

reading Neural networks and deep learning, having some difficulties already, but still able to follow, see if I could finish the first example.

when working with clojure on one of my old laptop, the executing speed is really slow. i turned on -verbose and it took a very long time to load all the libraries (on my old and slow laptop). it was just a simple script and won't need most of the libraries to be loaded.

there're tools to shrink library size, like ProGuard, but again, make them work with clojure gonna be very difficult.

there're also tools to reduce jvm startup time, drip, nailgun, etc.

drip is easy to use and uses fresh jvm, it enhanced about 50% startup time when I tested.

however, even re-use jvm, calling clojure.main to evaluate a script still quite slow, I guess there will be something to do for improvement, from the page I saw AOT all libraries actually just helped a little bit of the speed even though AOT is suggested, not worth the time to try making the script AOT for now.

found a useful package for emacs: clj-refactor.el

the demo gif is adding a core library clojure.string, I wonder how it works on other dependencies.

well, there is a new build/deployment tool: Habitat, by Chef

got attentions because it's written in rust

this is my frustration with all these containerization things. I'm frustrated because container is a good thing and we should adapt it. but new tool/framwork every month? (talking about those who did better work than existing ones).

I would rather spend time for LXC and read some docker source code.

anyways, I'm reading Puppet for Containerization, quite good, learned few new things.

first, you can define multiple vagrant machine in a yaml config file: see Multi-Machine Vagrant with YAML

it's a very simple way to create multi-nodes local environment.

then it's couple protocols, RAFT and Gossip

and the docker VXLAN-based overlay network and more details (require linux kernel >=3.19)

wait, what is VXLAN?

note, upgrade kernel from ubuntu trusty (3.13.0), install one of these packages:

using the latest 16.04 LTS is easier.

the stack of examples in the book also includes registrator for service registry, BIND for dns, consul for service discovery and key-value store, swarm for docker clustering, and of course, puppet for configuration management.

going through the examples is easy, but when applying to your own stack is a different story. it involves so many tools and digging deep into each of those area is a huge task.

the stask looks reasonable and this book helps a lot for getting a general idea of containerization. actually I'm more interested in this kind of DIY appoaches than kubernetes.

Thursday, June 16, 2016

Lamport timestamps is the lamport clock thing mentions in gossip protocol, for distributed message ordering.

the protocol also mentions the payload and entire message framing must fit within a single UDP packet

so I googled how large is a single UDP packet: The most reliable and efficient udp packet size?

it's the minimum MTU size that an host can set is 576 and IP header max size can be 60 bytes (508 = 576 MTU - 60 IP - 8 UDP)

576 bytes is the RFC791, MTU minimum packet size

1500 bytes is the MTU of Ethernet limit (some mathematics of Ethernet here)

so fit within a single UDP packet to avoid fragments: UDP Packet size and packet losses

some random reads:

a discussion on Why isn't there an open sourced Datomic?, brings out hitchhiker-tree: a datomic like database.

ClickHouse: a distributed column-oriented DBMS

Ancient code: programming on paper!

My First 10 Minutes On a Server - Primer for Securing Ubuntu, usually if this kind or article gets on frontpage, the comments are more interesting. discussion on hacker new and proggit.

learned couple things from the comments: Port knocking and Should I change the SSH port to < 1024?

Friday, June 17, 2016

did a super simple euro 2016 fixtures and group table page using this api

a simple function to convert UTC to lcoal timezone (+8)

(require '[clj-time.core :as t])
(require '[clj-time.format :as f])

(defn to-local-datetime [d df]
  (let [parsed-date (f/parse d)
                    my-timezone (t/time-zone-for-offset 8)
                    date-format (f/formatter-local df)]
    (f/unparse date-format
               (t/to-time-zone parsed-date my-timezone))))

(to-local-datetime "2016-06-17T21:03:57Z" "EEE MMM dd hh:mm")
;; "Sat Jun 18 05:03"

the free-keys emacs package lists out all unused keybindings (under current mode) for you.

I decided change my terminal escape key to M-[, since M-n is for auto-complete selection.

Saturday, June 18, 2016

quickly went through the rest of Puppet for Containerization

later chapters are examples of how to implement a specific setup, so lots of configuration codes and not much new things could be found. one example uses kubernetes, but a very brief introduction only.

the take away is there is docker and kubernetes modules for puppet.

continue reading Java Network Programming, since kotlin also comes with a repl (./bin/kotlinc-jvm), so it's quite useful to test out examples in the book (convertion from java codes is more straight forward than using clojure):

$ ./bin/kotlinc-jvm
Welcome to Kotlin version 1.0.2 (JRE 1.8.0_77-b03)
Type :help for help, :quit for quit
>>> import java.net.*
>>> InetAddress.getByName("www.google.com")
www.google.com/216.58.203.36
>>> InetAddress.getAllByName("www.google.com")
[Ljava.net.InetAddress;@11be3c5
>>> InetAddress.getAllByName("www.google.com").forEach(::println)
www.google.com/216.58.203.36
www.google.com/2404:6800:4005:805:0:0:0:2004
[kotlin.Unit, kotlin.Unit]
>>>

one interesting thing is InetAddress.isReachable():

>>> InetAddress.getByName("www.google.com").isReachable(1000)
false

from Problem with isReachable in InetAddress class, found that you actually need root privilege to run ping.

learned some history about zip and gzip: How are zlib, gzip and Zip related? What do they have in common and how are they different? - Stack Overflow

working with varnish you'll need to handle the Vary: Accept-Encoding header: you don't want different order of deflate and gzip values create another cache object variant. now I know they are wrappers of zlib, the library browser uses.

and I understand now why most of time you'll see smaller .tar.gz file than .zip: gzip applies deflate across all files while zip applies per file.

Tuesday, June 21, 2016

Dockercon 2016 is on, first news is Docker 1.12: Now with Built-in Orchestration! and their Docker Stacks and Distributed Application Bundles (DAB)

so it bundled load balancer, service discovery and key-value store, inner node communication using gRPC and http/2, IPVS for load balancing.

Raft consensus group from this paper: In Search of an Understandable Consensus Algorithm (pdf)

I don't see v1.12 on their xenial repository yet, but really want to try it, meanwhile reading some articles first:

trying other text-mode browsers, I just couldn't remember so many keys with w3m, I want to use arrow keys and menus only.

first I tried Links2. links2 -g starts a graphic mode browser, I like that. but lack of tab bothers me. so I tried another variant: Elinks, and switched to it completely.

comparing to w3m, elinks is better at:

one feature I missed is how easy that w3m sends current url to firefox. but elinks can do it easily too:

add these two lines to ~/.elinks/elinks.conf

set document.uri_passing.firefox = "firefox %c 2>/dev/null"
bind "main" "F10" = "frame-external-command"

sometimes I'll use emacs terminal mode, didn't work very well because so many keybindings are not working as expected under terminal mode, for example the esc key.

tonight I just found out that arrow keys and del key bring my evil-mode emacs-state to nomral-state, I don't know why but somehow useful to me.

Wednesday, June 22, 2016

another example to show how concise kotlin is: a daytime client, here is the java code (quite similar to the code in java network programming)

I tried rewriting it to kotlin:

import java.net.Socket

val stream = Socket("time-a.nist.gov", 13).getInputStream()
stream.reader().buffered().use {
  it.lines().forEach(::println)
}

on os x, need to compile elinks v0.12pre6 manually. v0.11 doesn't work, neither from brew nor compile.

another fast clojure repl solution, I combined it with drip, only needs 160ms to start a repl, no dependencies, but still cool.

Thursday, June 23, 2016

Eclipse Neon is released. I like the neon theme (good use of css box-shadow).

repl is nothing new when you working with clojure, even kotlin and java 9 have it. but this article: How I built my JAVA REPL is interesting to me. (the repo is here )

there is no dependency required, amazing.

JHipster generates spring boot and angularjs project, heard it from this video: Making Java Developers Hip Again

wanna find unicode characters that looks normal english letters? this is a list of confusable characters

after Elixir v1.3 released, Erlang/OTP 19.0 has been released as well. I really want to spend some time on erlang/elixir.

here's an interesting talk I found from the erlang/otp release thread: mnesia + leveldb: liberating mnesia from the limitations of DETS

LevelDB is not new, but I saw couple times from news recently, I decided to check it out:

this is the official documentation, there're two java wrappers: LevelDB JNI and LevelDB in Java

a hello world example (using LevelDB JNI library):

import org.iq80.leveldb.*;
import static org.fusesource.leveldbjni.JniDBFactory.*;
import java.io.*;

class LevelDB {
  public static void main(String[] args) throws IOException {
    Options options = new Options();
    options.createIfMissing(true);
    DB db = factory.open(new File("/home/ubuntu/example"), options);
    try {
      db.put(bytes("Tampa"), bytes("rocks"));
      String value = asString(db.get(bytes("Tampa")));
      System.out.println("got value: " + value);
      db.delete(bytes("Tampa"));
    } finally {
      db.close();
    }
  }
}

and Makefile:

compile:
	javac -cp leveldbjni-all-1.8.jar LevelDB.java 

run: 
	@java -cp .:leveldbjni-all-1.8.jar LevelDB	

clean:
	rm -f LevelDB.class

the leveldbjni-all-1.8.jar is just about 1MB, like a nosql version of sqlite

a full implementation build on top of leveldb is RocksDB by facebook ( more details )

when using ubuntu/xenial vagrant box, there is some problem with setting the private network. since 15.10 ubuntu changed tranditional network interface names (eth0, eth1 ..) to Predictable Network Interface Names

newer version of vagrant resolved this, a workaround is change the config.vm.network line to

config.vm.network "private_network", ip: "192.168.33.10", auto_config: false
config.vm.provision 'shell', inline: "ifconfig enp0s8 192.168.33.10 netmask 255.255.255.0"

ip link gives you all network interfaces in your machine.

npm semantic version calculator is a easy way to get correct version syntax for node.js package.

when bash-completion is activated, the filename completion does not work with some commands. there is another shortcut to force bash completes filename: M-/, more details here

Friday, June 24, 2016

Glowroot is an open source application performance management (APM) tool, easy to use. check the demo site

dropwizard metrics is another option, but need other tools for collecting data and dashboard.

rocksdb has another interesting usage - as mysql storage engine, this is the FOSDEM 16 talk: RocksDB Storage Engine for MySQL.

I think leveldb is good for persistent cache and persistent queue, quite interested in working on a simple version of them.

phoenix 1.2 has been released, I've decided I want to try it and will spend some time on it.

adding a memcached cache for the site, why not redis? I don't know.

memcached client for clojure, Spyglass seems the only choice.

it supports binary protocol (some brief introduction I found on google book )

adding it is quite easy, the more difficult problem always be cache invalidation.

I set a long ttl, I know what keys I need to purge when there are updates.

for deployment, I added a simple check to see whether remote git branch has new updates:

git remote update
if [ -n "$(git log HEAD..origin/master)" ]
then
    // deploy
    // purge keys
    // rebuild cache for those purged keys
fi

update value by cache key is not an option, could be done when I added a simple lambda service for storing functions.

an easier way is clear all on deployment: echo "flush_all" | nc memcache-host 11211 and rebuild some critical cache after purging.

Saturday, June 25, 2016

I'll dive into erlnag/elixir and spend rest of the year on it, as usual, starting with books:

first one is Programming Erlang by Joe Armstrong, he has a blog post about the book. my local library has first edition, I may take a look first.

second one is Learn You Some Erlang for Great Good!, LYAHFGG is a good book, so I think this one probably good as well.

local library also has another one: Introducing Erlang, I think I'll take this one too.

Sunday, June 26, 2016

did a SSL Server Test today and got an F

found that there is Yet Another Padding Oracle in OpenSSL CBC Ciphersuites

an apt-get update && apt-get upgrade fixed it.

actually I was checking nginx ssl config. I noticed server response time is slower via https comparing to access directly to backend. I didn't expect ssl termination costs so much time, I tried to see is there anything could help from tuning nginx config, found nothing yet.

but I had a long ssl_ciphers list, so did a little bit tidy up:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
ssl_prefer_server_ciphers on;

start reading Learn You Some Erlang for Great Good!, notes will be uploaded here

Monday, June 27, 2016

I'm looking for some cache service implemented by java, found JCS and Ehcache, Ehcache also on the list of awesome java, under distributed applications section, I may check it out first.

also interested in another library under the section: hazelcast, an in-memory data grid.

clojure also has an awesome list, I'm checking with liberator, a library for building restful api. their decision graph (svg) is quite useful even for implementing an api library yourself.

a simple testing app is like: 

;;[liberator "0.14.1"]
;;[ring/ring-core "1.5.0"]
;;[ring/ring-jetty-adapter "1.5.0"]
;;[compojure "1.5.1"]

(ns libera
  (:require [liberator.core :refer [resource defresource]]
            [ring.middleware.params :refer [wrap-params]]
            [ring.adapter.jetty :refer [run-jetty]]
            [compojure.core :refer [defroutes ANY]]))

(defresource parameter [txt]
  :available-media-types ["text/plain"]
  :handle-ok (fn [_] (format "The text is %s" txt)))

(defroutes app
  (ANY "/foo" [] (resource :available-media-types ["text/html"]
                           :handle-ok (fn [ctx]
                                        (format "<html>It's %d milliseconds since the beginning of the epoch."
                                                (System/currentTimeMillis)))))
  (ANY "/bar/:txt" [txt] (parameter txt))
  (ANY "/" [] (resource)))

(def handler 
  (-> app 
      wrap-params))

(run-jetty handler {:port 3000})

Thursday, June 30, 2016

some ssl cert verify commands:

# check
openssl x509 -in cert.crt -text -noout

# verify
openssl x509 -noout -modulus -in cert.crt | openssl md5
openssl rsa -noout -modulus -in private.key | openssl md5

# convert to pem
openssl x509 -inform PEM -in cert.crt > cert.pem
openssl rsa -in private.key -text > private.pem

using groff to write resume.

-t to preprocess with tbl -fH to use helvetica fonts (-fT is times family) -Tascii outputs ascii text

inside the document, use .fam H to switch fonts to helvetica

grog helps you determine which preprocessors/macros are required. useful when you copy-paste an example from google search result but don't know which preprocessors should use.

$ grog sample.groff
groff -t -ms sample.groff

my last day of this job, it's been 6 years and time for a change.

one good practice I've learned when doing infrastructure is try to make things disposable, so I disposed myself as well.

Blog Archive