Jim Cheung

Saturday, October 01, 2016

when I was working on a CodeIgniter project, I wanted to reload my page once I updated files.

I think monitor files and reload page on a certain event is quite simple. so I decided to do it myself.

the idea is use a websocket server, pushes file change events to javascript client side, and let javascript side to reload the page.

I've done the poc and put it on github.com/usrjim/ws-reload.

reading Web Development with Go: Building Scalable Web Apps and RESTful Services, borrowed from local library.

It's quite simple and almost finished.

at the same time, reading little bit of Programming in Lua everyday.

lua is something between php and go, also very simple. I kinda like it and want to use it for shell scripting.

a nice comparison with AWS and GCE (source):

  • AWS has 2 storage solutions with different APIS: S3 and Glacier; Compare that to Google. Just one storage solution to serve all needs. You get a backed in CDN for free!

  • AWS has two queuing systems (SQS and Kinesis) and still require the developer / admin to adjust the scaling of infrastructure. Google has just one Pub/Sub. You get push notifications on top. No need to tune knobs to get extra scale. It just works.

  • AWS load balancers and persistent disks need warming up before high usage. If you are running a website on global scale, you need to use DNS geo load balancing on top. Google load balancers are global (as opposed AWS regional load balancers), no need of DNS tricks. No need of prewarming. Google persistent disks need no prewarming. You can mount a single persistent disk on multiple instance and share data easily.

  • Security: Google encrypts data at rest and at wire by default. Try doing that on AWS. Google takes care of SSH key provisioning and management. AWS: You have to do it by yourself.

  • AWS NATs and micro instance are known to be unreliable. Google has live migration. If something goes wrong with instance they work their magic behind the scenes so that you don't have to worry about migrating the instance to another physical host.

  • Automation: Instance id are not global on AWS. Have fun creating maps and stuff inside CloudFormation templates. Google Cloud resources are global. All resources (images ids) have a global identifier. No more messing with zonal vs regional vs global resources.

I didn't pay much attention to Live Migration before, it seems pretty cool.

I picked a few talks from GOTO Chicago 2016 to watch, this one Providing Flexible Database Consistency Levels with Manhattan at Twitter • Boaz Avital is really good.

It's funny that much like concurrent programming, use a channel to force data comes in and out synchronously. I'm more interested in how they maintain the log.

it's the first time that I heard about Apache BookKeeper, looks like a lower level version of kafka.

another one may be related is Distributed Commit Log: Application Techniques for Transaction Processing by David McNeil from Strange Loop 2016

Wednesday, October 05, 2016

repeats/Repeat: Repeat yourself with some intelligence. is a java program to record and playback your mouse actions.

on searching a simple javascript template library, I found mustache and mustache.js. it's logicless, there is no if, and found an implementation for lua too: Lustache

a great hit recently: How it feels to learn JavaScript in 2016

I recently tried some clojure frontend library and react, I felt the same.

now I just code es5 with createElement / createClass to learn react, I don't miss jquery only by this way.

my ws-reload is working, not good by it's working really well.

now I'm using

ssh-keygen -o -a 100 -t ed25519

to generate my ssh keys according to this article: Upgrade your SSH keys!

ed25519 key is so short! but it's more secure: ecc - Is it bad that my ed25519 key is so short compared to a RSA key?

more about ed25519: Ed25519: high-speed high-security signatures: Papers

creator of zeromq, Pieter Hintjens has passed away. his books and slides

one good feature of go 1.7.1: HTTP Tracing

httpstat is a tool implemented with it.

then I found out digitalocean's name servers are really slow. I changed back to cloudflare, response time is much better now.

Thursday, October 06, 2016

about clojure, I have a list of common dependencies for new projects. lein new only creates a minimal project.clj with no dependencies.

create your own lein template can address this problem.

first run:

$ lein new template mytemplate

then edit src/leiningen/new/mytemplate.clj, add project.clj to (->files ... block, like this:

   (->files data
             ["project.clj" (render "project.clj")]
             ["src/{{sanitized}}/foo.clj" (render "foo.clj" data)])

create resources/leiningen/new/mytemplate/project.clj with all the dependencies. (also add any .clj files you want).

run

$ lein install

to install it locally. update these to your home directory ~/.lein/profiles.clj:

{:user {:plugins [[lein-ancient "0.6.10"]
                  [mytemplate/lein-template "0.1.0-SNAPSHOT"]]}}

(mytemplate/lein-template "0.1.0-SNAPSHOT" is the namespace declared in mytemplate/project.clj)

(lein-ancient is a good plugin to update your dependencies all at once, very useful)

now I can create new project with:

$ lein new mytemplate myproject

Friday, October 07, 2016

I like to listen music on youtube, usually I just need the audio part.

mpv is perfect for this.

first apt-get install mpv or brew install mpv (os x), then simply

$ mpv --no-video https://www.youtube.com/watch?v=fQ8ze1-LCrg

to play a playlist, use

$ mpv --no-video https://www.youtube.com/watch?list=PL628B1FFF96678C60

and some shortcuts:

| >     | next            |
| enter | next            |
| <     | previous        |
| p     | pause           |
| m     | mute            |
| 9     | decrease volume |
| 0     | increase volume |
| {     | play slowly     |
| [     | play slowly     |
| }     | play faster     |
| ]     | play faster     |
| q     | quit            |

when writing simple scripts (lua more specifically), can use watch to do auto reload:

$ watch -n 1 -c -d lua myscript.lua

I like watch shows difference compared with last result and it keeps results in a fixed position, I don't need to trace the tail of the output.

table is the only data structure in lua. it's like php's array, can be an indexed or associative.

however, you can't sort an associative table easily. compare function applied to table.sort() in lua only has value as input, but you need the key.

the solution is use a temp table:

local tbl = {
  f = "foo",
  b = "bar",
  z = "zoo",
  c = "cat",
}

local tmptbl = {}
for k, v in pairs(tbl) do
   table.insert(tmptbl, {x=k, y=v})
end
table.sort(tmptbl, function (a, b) return a.x < b.x end)

for _, s in pairs(tmptbl) do
  print(s.y)
end

however, I still like the simplicity of lua, the dynamic version of go.

all my services running in docker containers, but maintain those images are not as easy as apt-get upgrade

I'm moving stable services out of containers and only use docker to host experimental components. this makes things so much easier and simple.

then I found out nginx v1.10.0 was released 5 months ago, the stable version includes http/2 and dynamic modules

also setup varnish v5.0, it's faster than application level cache and easier for me than setup cache using nginx

varnish is a good piece of software, basically if you want to implement a cache server yourself, you'll end up with one as same as varnish

the .deb file is just 2.5M. I'm glad that I spent time with varnish before, when you understand it, it could be so flexible and performs well.

Saturday, October 08, 2016

tried google's new noto mono font, pretty good. even clearer than roboto mono, nice.

netbeans 8.2 has been released, includes support for multiple carets, php7, es6, jsx (react), docker ... details check 8.2 new and noteworthy page

I've been waiting for multiple carets for a long time, now I don't need emacs for multiple carets.

I also found there's a puppet plugin in verfied channel, I'll need that.

netbeans will be under apache soon, I hope it works wel, it's still my favourite editor among itellij, emacs, vim, atom, sublime, vs code, light table.

Monday, October 10, 2016

LuaRocks is the package manager for lua, also I can check what's the most popular packages on the website. that is very helpful.

now I'm playing with lua-term.

one library that I'm interested in but not listed under luarocks is luafun, a high-performance functional programming library for Lua.

I want to try lua on nginx and haproxy first.

I compared go and lua version of Skynet 1M threads microbenchmark, lua 5.2 is already faster than go 1.7.1 on my laptop, luajit 2.04 is 50% faster than lua 5.2 version, if I appended GOMAXPROCS=1, the luajit result is 5 times faster than the go one, amazing.

however, it brings up a question about lua, how to use multi-cores?

Lua Lanes and lua-llthreads are the two people recommend.

there is a new clojure book: Professional Clojure, I downloaded the sample chapter and read it. it's good and I will see when I have time to read the book.

Doug Hoyte (author of Let Over Lambda) has some interesting articles about nmap,

I also borrowed Let Over Lambda from local library, not an easy book, but I hope I can learn something from it.

Wednesday, October 12, 2016

to install luarocks packages locally, use

$ luarocks install --local lua-term

or use custom directory: 

$ luarocks install --tree lua_modules lua-term

then use

$ eval `luarocks path`

to export LUA_PATH and LUA_CPATH environment variables

to make it permanent, append to .bashrc:

$ luarocks path >> .bashrc

The Best Lua Web Frameworks covers some popular web frameworks.

I'll try Lapis first, since it's quite different from stuffs I've been using.

while reading the lua book, found this section is interesting: The Semantics of the Generic for

usually I don't pay much attention to the for loop, not even go. but this section does a good job explaining it.

for simple loop:

for k, v in pairs(t) do
  print(k, v)
end

can also write:

for k, v in next, t do
  print(k, v)
end

some good reads:

The Rise and Fall of Scala, I learned some scala before, I really don't like it.

But I want to try akka, I hope kotlin can have a interesting framework like akka and play.

these two books are about scala too: Functional and Reactive Domain Modeling and Reactive Web Applications.

Friday, October 14, 2016

I'm reading ØMQ - The Guide (zguide), lua edition.

but lua-zmq seems not working with the latest version of zero mq.

I changed to use lzmq, which also provides example of zguide

to build luarocks using luajit:

$ ./configure --with-lua=/usr/local/share/luajit-2.0.4 --lua-suffix=jit --with-lua-include=/usr/local/include/luajit-2.0 --with-lua=/usr/local

when I working on the pub-sub example, I want to monitor the packets between server and client.

using tshark to follow tcp stream:

$ tshark -i any -f "tcp port 5566" -z follow,tcp,ascii,1

but there's another cool tool for this:

$ tcpflow -i any -c port 5566

also play a little bit of core.async, this snippet will make n requests to the target web server, simply tail the access log to see those requests.

(defn req [x] 
  (:request-time 
   (curl/get "http://192.168.33.20" {:headers {:user-agent "-"
                                               :referer x}})))

(defn launch [n]
  (a/<!! (->> (range n)
              (map #(a/go (req %)))
              a/merge
              (a/into []))))

I also like this example from Mastering Concurrent Processes with core.async | Clojure for the Brave and True:

(defn upper-caser
  [in]
  (let [out (chan)]
    (go (while true (>! out (clojure.string/upper-case (<! in)))))
    out))

(defn reverser
  [in]
  (let [out (chan)]
    (go (while true (>! out (clojure.string/reverse (<! in)))))
    out))

(defn printer
  [in]
  (go (while true (println (<! in)))))

(def in-chan (chan))
(def upper-caser-out (upper-caser in-chan))
(def reverser-out (reverser upper-caser-out))
(printer reverser-out)

(>!! in-chan "redrum")
; => MURDER

(>!! in-chan "repaid")
; => DIAPER

Saturday, October 15, 2016

besides zero mq, there is nanomsg, they even have a page for comparison with zero mq

(nanomsg is written by one of zero mq author: Martin Sustrik.)

there is also a package for lua: nanomsg/luajit-nanomsg

for Protocol Buffers in lua, can try google/upb

re-visit the core.async walkthrough, understand more about core.async

use thread when you performs a blocking call (just like future, but returns a channel):

(let [c (chan)]
  (thread (>!! c "hello"))
  (assert (= "hello" (<!! c)))
(close! c))

alt!! with dotimes is just like wait group in go, it takes whatever returns the first, by looping with dotimes, eventually it waits all go blocks returned their results:

(let [n 1000
      cs (repeatedly n chan)
      begin (System/currentTimeMillis)]
  (doseq [c cs] (go (>! c "hi")))
  (dotimes [i n]
    (let [[v c] (alts!! cs)]
      (assert (= "hi" v))))
(println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms"))

the walkthrough doesn't cover alt! / alt!!, this stackoverflow question has more details: how to understand alt in clojure core.async - Stack Overflow

also read the examples from the documentation:

(go 
 (let [timeout-ch (timeout 1000)
       trade 100]
   (->
    (alt!
     [[trade-ch trade]] :sent
     timeout-ch :timed-out)
    print)))

note here, [[trade-ch trade]] means put trade to channel trade-ch (blocking).

puts must be in nested vector, otherwise it was treated as a vector of multiple channels.

lua also has a package for CSP: loyso/LuaCSP: Communicating Sequential Processes in Lua

using tmux, can easily run a program in a new pane:

$ tmux split-window "command"

pane will be closed/exit once the command finished.

set remain-on-exit option can resolve this:

$ tmux set-option remain-on-exit on

there is also another way, append ; read to the end of command:

$ tmux split-window "command; read"

read waits for user input, the pane will be kept until you press enter (newline)

even you started a long-running process in foreground:

Sunday, October 16, 2016

when I told my colleague I'm learning lua, he suggested me to try awesome window manager, it's written in lua

I tried and I love it! it's so fast~

not easy to use though (well, not as hard as the old days)

I decided to dive deep into it. notes will be updated to my awesome wm page.

Tuesday, October 18, 2016

under awesome wm, some keyboard tweaks need to do it manually:

disable touchpad tap:

$ synclient MaxTapTime=0

make caps lock as ctrl:

$ setxkbmap -option 'caps:ctrl_modifier'

to make them permanent, add to ~/.profile.

I used amethyst on os x before, but didn't find it useful.

until I tried awesome wm, now I get the idea how to manage windows.

I'm using an external keyboard, so:

mod1 = alt + shift
mod2 = ctrl + alt + shift

the commands that I found useful are:

switch to full screen layout and switch back

mod1 + space | cycle to next layout
mod2 + space | cycle to previous layout

cycle windows

mod1 + j - focus the next window counterclockwise
mod1 + k - focus the next window clockwise

swap windows position

mod2 + j - move the focused window one space counterclockwise
mod2 + k - move the focused window one space clockwise

resize windows

mod1 + h - shrink the main pane
mod1 + l - expand the main pane

un-float / float window

mod1 + t - toggle whether or not the focused window is floating

disable / enable amethyst

mod2 + t - toggle globally whether or not Amethyst tiles windows

move window to another space

mod2 + h

Thursday, October 20, 2016

there're so many php 7 books coming out. I'm reading PHP 7 Programming Blueprints.

the book is not as good as its TOC. it covers some interesting topics: websockets, PEG, zeromq, reactive.

but when I read the book I found it actually pretty bad. like this: 

$q = " INSERT INTO " . $this->table . " VALUES ( '".$values['name']."', '".$values['age'] . "' ,'".$values['country']. "')";

it's 2016, I can't believe they still teaching people using this way to write SQL. (and they don't even mention PDO)

and docker, some node and python tools ... definitely one of the worst books I've read.

I have difficulties with git diff, I always don't know how to read the diff chunks. finally I decided to read more about it:

so there are

(master) $ git diff dev
(master) $ git diff dev..master
(master) $ git diff master..dev

the later two are easy, - is the left hand side, + is the right hand side.

I can't remember the first one: git diff dev is same as git diff master..dev:

diff --git a/bar/main.js b/bar/main.js
index 43eaecf..9211428 100644
--- a/bar/main.js
+++ b/bar/main.js
@@ -1,6 +1,7 @@
 var foo = "bar";

-console.log(foo);
+//console.log(foo);
+
+var foo = "jar";

-console.log(foo + 1);

there're a and b, - and +

before I thought that - as deletion and + as added lines, but this is confusing sometmes. the better way is think - as changes of left hand side (a) and + as changes of right hand side (b).

and there are hashes for a and b, you can check hashes by:

$ git rev-parse master:bar/main.js
$ git rev-parse dev:bar/main.js

to check which commits contain the file (blob):

git rev-list --all |
while read commit; do
    if git ls-tree -r $commit | grep -q 9211428; then
        echo $commit
    fi
done

to check which branch contains the commit:

$ git branch --contains d7d8fafce511973b566547b616b4b7db49364f47

now check difference between branches by commits:

(master) $ git cherry dev

above is same as $ git cherry dev master (reverse order or diff ... WTF)

to cherry-pick a range of commits:

$ git cherry-pick A^..B

A^ means include commit A as well, A..B will not include commit A.

another powerful command I found in the git help cherry-pick:

git rev-list --reverse dev -- bar/main.js | git cherry-pick --stdin -n

all commits involves bar/main.js will be picked to current branch

Wednesday, October 26, 2016

borrowed Monitoring with Ganglia from local library, but didn't find it useful.

did some research online, looks like collectd for collecting data, statsd for aggregation, or persist to Graphite or InfluxDB, and use Grafana as dashboard.

what I'm interested in is writing collectd data to Riemann, there's Plugin:Write Riemann for this.

on ubuntu, collectd has too many dependencies, I installed collectd-core instead: (graphite-carbon receives metrics and writes them to disk storage)

$ sudo apt-get install collectd-core graphite-carbon

then create /etc/collectd/collectd.conf:

$ sudo vi /etc/collectd/collectd.conf

to test, just test get cpu metrics every second, save to csv and POST to a remote http server:

Interval 1
Hostname "vagrant-ubuntu-trusty-64"
LoadPlugin cpu
LoadPlugin write_http
LoadPlugin csv

<Plugin "write_http">
  URL "http://192.168.33.1:8888/x"
</Plugin>

<Plugin "csv">
  DataDir "/var/lib/collectd/csv"
  StoreRates true
</Plugin>

write to graphite, use the Plugin:Write Graphite

that's just the first step, still need to find more reference on how to process metrics collected from collectd using riemann.

saw vue.js few times on my dev news, didn't pay much attention before since author is a chinese. I always kept my distance from chinese developers.

one of my colleague built something with vue.js, and it looks very simple comparing react and angular, also seems gitlab is using it: Why We Chose Vue.js and quite positive on hackernews

why I still prefer working with jquery is that it doesn't require a built process, setup a simple react hello world app requires thousand of js files fetched by npm.

vue.js doesn't require this, include it in html and starting coding, just like jquery. I'm not a frontend developer, this is more than enough for me.

in their tutorial, found two essential js libraries:

and this nice article: Debouncing and Throttling Explained Through Examples

one nice thing about tmux is that you can send commands to a specific pane (in a specific window) anywhere, when I playing songs in one pane, if I want to pause it:

$ tmux send-keys -t 1.3 p

1.3 means window 1, pane 3, tmux ls gives you window list, and tmux display-panes shows your pane numbers.

you can also get contents from any pane:

$ tmux capture-pane -p -t 1.3

-p means output to stdout, can also specify starting and ending lines by -S and -E, but I found it's difficult to use.

man tmux for more details.

I'm reading Pro Spark Streaming, examples are written in scala, but I don't like scala ...

found two wrapper for clojure:

now I feel better and I can continue with the book...

Thursday, October 27, 2016

I want to play a little bit of terminal ui, people said termbox is better than ncurses

was going to try with lua, but there're so little reference, termfx already the best I could find.

on the other hand, go has much more examples: from the original termbox-go, to gocui and termui.

I think I'll start with go ...

before I thought object size limit for memcached is 1MB, today my colleague points out you can set up to 128MB with option -I, read memcached -h for more details.

Blog Archive