Jim Cheung

Wednesday, November 02, 2016

an interesting article: scripting in Clojure using boot

as we know boot can do shebang

but this article is about using boot as a console framework to build custom tasks, actually quite useful.

I decided to study more about boot.

and one tutorial I'm reading is using boot as well: modern-cljs: A series of tutorials on ClojureScript

one thing I want to do is like start a repl with some dependencies, but no need to create a project.

using boot is quite easy:

boot -d org.clojure/core.async:0.2.395 repl

also there're some JVM Options can be tuned for a faster startup time.

Thursday, November 03, 2016

lein-skummet is another option to reduce jvm startup time.

to use it, simply put these lines to your project.clj:

:dependencies [[org.skummet/clojure "1.7.0-r2"]]
:exclusions [[org.clojure/clojure]]
:profiles {:default []}
:plugins [[org.skummet/lein-skummet "0.2.2"]]

(note, it replaces org.clojure/clojure "1.8.0" with org.skummet/clojure "1.7.0-r2")

then these commands are available:

$ lein skummet compile
$ lein skummet run
$ lein skummet jar

or 

$ lein do skummet compile, skummet run

I tested it, seems not all libraries are compatible with it, for example clj-http causes compile error, switched to http-kit compiles normally.

the output does run faster, maybe helpful.

Friday, November 04, 2016

about lua

Torch looks interesting too, machine learning in luajit.

reading Mastering Clojure, most of clojure books don't cover core.async, this book probably has the most detailed chapter about core.async.

it also covers the actor model with pulsar

still studying apache spark, I like spark shell, a repl to do data analysis.

too bad it supports scala and python only, I may need to re-visit my scala notes ..

some random notes:

Saturday, November 05, 2016

another docker story: Docker in Production: A History of Failure

I still think container is the future, but not docker anymore. I prefer a more boring solution for containers.

Monday, November 07, 2016

to create a fatjar for a java library with gradle, add these lines to build.gradle:

task fatJar(type: Jar) {
  baseName = project.name + '-all'
  from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
  with jar
}

then run:

$ gradle fatjar

to include java source in a clojure project, add this line to project.clj:

:java-source-paths ["src/main/java"]

this works with uberjar as well.

when doing :aot :all, uberjar will evaluate variables, use a delay is one workaround that I like:

(def service (delay (get-service)))

not only it solves the problem of uberjar, but also @service is more readable than service.

just like dig is modern version of nslookup, ss is modern version of netstat, man ss for more details, -tupln also works on ss:

$ ss -tupln

check existing connections: 

$ ss -t4n state established

or query by dst / src:

$ ss -t4 src xxx.xxx.xxx.xxx:22

to empty a file, usually will be done by $ cat /dev/null > somefile

there's another way:

$ truncate --size 0 somefile

mysqltuner is a useful tool to get recommendations for mysql config: my.cnf, just do

$ apt-get install mysqltuner

Thursday, November 10, 2016

another container management tool: dc/os

looks like it's built on top of mesos

at the comment part of this article: Running Online Services at Riot: Part II | Riot Games Engineering, author gave comparison on dc/os and kubernetes.

and finally I see cool aws lambda alternative: Serverless computing on DC/OS with Galactic Fog

gestalt is a framework for building microservice platform.

may need to spend some time on these because they looks pretty nice.

Kotlin 1.0.5 is here, I didn't write much kotlin recently. I only do kotlin when clojure is not allowed.

Five Neglected Computer Science Classics

nice list but those books probably too much maths to me, maybe I will try The Implementation of Functional Programming Languages.

still reading the zguide, it's a very good guide for common messaging patterns, well explained and good advices. a must read even though you are not going to use zero mq.

Saturday, November 12, 2016

a snippet to n go blocks, and collect all the results:

(defn dispatch [data]
  (let [n (count data)
        cs (repeatedly n a/chan)]
    (doseq [[v c] (map list data cs)]
      (a/go (a/>! c (do-something-with v))))
    (dotimes [_ n]
      (let [[v c] (a/alts!! cs)]
        (println v)))))

manually assign n to a smaller value if data size is too large.

Tuesday, November 15, 2016

looking for a simple file based key-value storage, MapDB is one of the options. but it lacks of clojure wrapper, I choosed Factual/clj-leveldb: Clojure bindings for LevelDB, which is using fusesource/leveldbjni: A Java Native Interface to LevelDB underneath.

there's another interesting one: saolsen/steveskeys: Key Value Storage In Clojure.

reading some old issues of Linux Journal, found some interesting projects/tools:

Wednesday, November 16, 2016

found a pretty theme for emacs: Subatomic Theme

Saturday, November 19, 2016

this week has been working a lot on clojure, finally decided to study paredit all over again:

move parentheses

| action          | function                    | hot-key  | alt-hot-key          |
|-----------------+-----------------------------+----------+----------------------|
| move ( to left  | paredit-backward-slurp-sexp | ctrl + ( | ctrl + alt + left    |
| move ( to right | paredit-backward-barf-sexp  | ctrl + { | ctrl + alt + right   |
|-----------------+-----------------------------+----------+----------------------|
| move ) to left  | paredit-forward-barf-sexp   | ctrl + } | ctrl + shift + left  |
| move ) to right | paredit-forward-slurp-sexp  | ctrl + ) | ctrl + shift + right |


move cursor

| action              | function             | hot-key        | alt-hot-key       |
|---------------------+----------------------+----------------+-------------------|
| move up one level   | paredit-backward-up  | ctrl + alt + u | ctrl + alt + up   |
| move down one level | paredit-forward-down | ctrl + alt + d | ctrl + alt + down |
| move forward        | paredit-forward      | ctrl + alt + f |                   |
| move backward       | paredit-backward     | ctrl + alt + b |                   |


select sexps

| action          | hot-key                |
|-----------------+------------------------|
| select forward  | ctrl + shift + alt + f |
| select backward | ctrl + shift + alt + b |


kill parentheses

| action           | function            | hot-key |
|------------------+---------------------+---------|
| kill parentheses | paredit-splice-sexp | alt + s |
|                  |                     |         |


add parentheses

| action      | function           | hot-key |
|-------------+--------------------+---------|
| wrap around | paredit-wrap-round | alt + ( |
|             |                    |         |


kill

| action        | function                             | hot-key              |
|---------------+--------------------------------------+----------------------|
| kill sexp     | kill-sexp                            | ctrl + alt + k       |
| kill line     | paredit-kill                         | ctrl + k             |
| kill backward | paredit-splice-sexp-killing-backward | alt + up             |
| kill forward  | paredit-splice-sexp-killing-forward  | alt + down           |
| raise sexp    | paredit-raise-sexp                   | alt + r              |
| join          | paredit-join-sexps                   | alt + J (upper case) |


re-indent

| action   | function               | hot-key |
|----------+------------------------+---------|
| reindent | paredit-reindent-defun | alt + q |

Tuesday, November 22, 2016

I had a bash script for connecting my hosts easily by selecting from a list, but it was every difficult to maintain. I rewrite it with lua:

#!/usr/bin/env luajit

local hosts = {}
local c = assert(io.popen("grep '#put' ~/.ssh/config", "r"))
for l in c:lines() do
   _, _, h = string.find(l, "Host%s+(%w+)%s+#put")
   table.insert(hosts, h)
end
c:close()
table.sort(hosts)

for i, s in pairs(hosts) do
  print(i, "-", s)
end
io.write("pick one: "); 

local pick = io.read()
local pickkey = tonumber(pick)
if pick == nil or pick == "" then
  print("abort")
elseif hosts[pickkey] then
  print("connecting", hosts[pickkey])
  os.execute("ssh " .. hosts[pickkey])
else 
  print("don't un")
end

to use just append #put at the end Host of line, like:

Host devhost #put

then devhost will appear on the list for selection.

playing aws lambda with clojure

java probably not a good fit for lambda due to its long startup time and memory requirement.

but I still prefer clojure (or even java) than node and python.

serverless is the new trend, but I'm still not sure it's better than tranditional stacks yet.

further reading: ServerlessConf slides

found another nice theme for emacs: Dracula theme

Thursday, November 24, 2016

TIL, if you create a git-hello command and put in your $PATH, you can trigger it by git hello

learned it from here: anvaka/git-also: For a file in your git repository, prints other files that are most often committed together

there're some other similar tools:

some noteworthy links:

Friday, November 25, 2016

CiteSeerX is a good place to look for papers, it provides cached version of the paper, very convenient.

video of Joe Armstrong interviews Alan Kay on CodeMesh

also Joe's keynote: Distributed Jamming with Sonic Pi and Erlang

finally there is a scripting solution for clojure/clojurescript: anmonteiro/lumo: Fast, cross-platform, standalone ClojureScript REPL

it's blazing fast and surely capable for scripting.

yogthos provided an example on reddit thread:

(def fs (js/require "fs"))

(defn ls [dir]
  (js->clj (.readdirSync fs dir)))

(let [args (-> js/process .-argv js->clj)
      dir  (last args)]
  (println "found files:" (ls dir))
  (println (.readFileSync fs "hello.cljs" "utf8")))

it calls nodejs dependency fs, another interesting point. now you can use clojure/clojurescript libraies and nodejs libraries, how nice.

for aws lambda with clojure, use uswitch/lambada: A more passionate way to write AWS Lambda functions, it's way more easy than following Writing AWS Lambda Functions in Clojure | AWS Compute Blog.

the speed is actually good. it was slow for this first run (2s for a simple task), but repeated runs are fast (under 100ms to few hundred ms).

deployment is fast, just uberjar and upload.

essentially it's a zip file and it will be extracted on ec2 managed by aws, which means you can pack files in the zip/jar, then read them from your function.

logs are a bit tricky, I still don't understand how aws events group lambda logs into a log stream. I can't find a way to set log stream name pattern, need to aws describe-log-streams to get all stream names.

I use aws logs --filter-pattern for now.

also 128MB for java fails a lot. raise to 512MB it becomes more stable.

another tip is you can view all sample input from different aws services on aws lambda console, when you click test your function. quite useful to get a general idea about which service provides what kind of data to the lambda function.

you can't pack amazonica, it's over the 50MB size limit.

I don't have good believe in serverless, but for some cases aws lambda could be helpful and more easy to manage.

in general aws lambda should be used when:

since input and output are aws services, and lambda is so tiny and there will be lots of them. good management on rols and policies are required.

Dply provides clould server free for 2 hours. you just need a github account (just use your github key to access).

I tried and it works well. good for testing.

for this kind of temporary servers, you don't want to add them to your known_hosts, append -o "UserKnownHostsFile /dev/null" option to ssh command.

(add -o StrictHostKeychecking=no to skip warning)

two hours actully is good enough for a paring section.

Saturday, November 26, 2016

I heard about Nightlight few times, finally tried it today.

you'll have an editor in browser with repl support when you run the program. maybe useful in some ways.

reading Professional Clojure.

after done couple clojure projects, I decided to go back to read more clojure books. I often found there should be better way to write clojure codes than what I've done.

a quick test on writing a http server using clojurescript with nodejs http module, runs with lumo:

#!/usr/local/bin/lumo -q

(def http (js/require "http"))

(defn handler [req res]
  (doto res
    (.writeHead 200 {"Content-type" "text/plain"})
    (.end "roarrrrrr!")))
    
(defn start-server []
  (-> (.createServer http handler)
      (.listen 3000 "127.0.0.1"))
  (println "server running at http://127.0.0.1:3000/"))

(start-server)

I can use clojure + java or clojurescript + javascript + node.js in one language, it's very cool.

but I need to re-visit my node.js books ..

Tuesday, November 29, 2016

Factual/durable-queue: a disk-backed queue for clojure, it's pretty simple and data is presisted in disk.

tried adamtornhill/code-maat: A command line tool to mine and analyze data from version-control systems, it's written in clojure. didn't pay attention to its book: Your Code as a Crime Scene. not found anything interesting yet.

love Dply, I wrote a simple webservice and put register command in the user script, my ssh connect script will fetch ip from the webservice.

now every time I start a dyln 2 hours free vm, I can use the same command to ssh in.

React Native Express is a good resource to learn react native.

I tried, but it's so complicated to create a project even with a boilerplate tool.

Wednesday, November 30, 2016

matryer/bitbar is a very useful tool to make some shortcuts with script and put it to os x menu bar.

start mpv with --input-ipc-server=/tmp/mpv-sock, you can send commands to control the player like:

$ echo cycle pause | nc -U /tmp/mpv-sock

nc -U is for unix socket, if unsupport can use socat.

other common commands are:

quit, playlist_next, playlist_prev, can also add file to list via loadfile $file append-play

combine with bitbar, I can control mpv anywhwere via menu bar.

I love bookmarklets, so this project is useful for me adzerk-oss/boot-bookmarklet: A Boot task for generating bookmarklets from ClojureScript namespaces

Blog Archive