Jim Cheung

Friday, March 03, 2017

finished Programming Elixir 1.2

the otp part is very basic, I actually want to know more about different kind of supervisor behaviours.

elixir is nice but I still prefer clojure for now.

funny that I saw this post on reddit: Elixir or Clojure?, it's a good summary of two languages.

I like ruby has some special options for reading from stdin:

-n will wrap your codes with while gets(); ... end, for example:

$ cat /etc/passwd | ruby -ne 'puts $_.split(":").first'

-a autosplits $_ to $F, use -F to specify pattern:

$ cat /etc/passwd | ruby -F: -ane 'puts $F.first'

-p is like -n, but puts $_ autometically:

$ cat /etc/passwd | ruby -F: -ape '$_="#{$F.first}\n"'

-r to require library

reading Text Processing with Ruby

reading file with ruby is simple and easy, much better than php's syntax.

for reading large file, use mmap, this is a good tutorial for it: Java NIO - Memory-Mapped Files with MappedByteBuffer

clojure has thebusby/iota library to deal with huge files.

reading Clojure Programming Cookbook too.

I found myself know too little clojure, this book taught me a lot.

Specter - Clojure's missing piece hits 1.0, looks powerful but I haven't try yet.

my current project needs to parse a big nested json, I think specter can help.

halgari/odin: An embedded extensible logic DSL for Clojure. is another library may be useful for me.

docker, my colleage suggests me Kitematic, a GUI tool for docker.

it's so easy to manage containers with kitematic, if image's dockerfile has VOLUME mount point, you can map a local directory via GUI. but if not, you'll need to rebuild it or enter the VM to start the container manually with -v option.

overall it's an excellent tool for local development with docker.

Saturday, March 04, 2017

I'll rewrite dev news page in order to make it faster.

I also want deep dive into redis and postgresql, not just basic, more in-depth study.

will start from redis and go through their Redis documentation page, it has lots of details.

Monday, March 06, 2017

read gz files inside irb:

irb> require 'zlib'
irb> Zlib::GzipReader.open("access.log.2.gz") { |z| z.readlines.grep(/assets/).map{|x| x.split('"')[1]} }

ruby provides richer funcions to parse/filter data, and repl can save previous results.

Tuesday, March 07, 2017

aws s3 was down in us-east region, it reminds people s3 is the biggest single point of failure.

found an interesting implementation from this comment, regarding Dodging S3 Downtime With Nginx and HAProxy:

Heads up a simple yet production ready NGINX location block to proxy to a public s3 bucket looks like: 

# matches /s3/*
location ~* /s3/(.+)$ {
    set $s3_host 's3-us-west-2.amazonaws.com';
    set $s3_bucket 'somebucketname'

    proxy_http_version 1.1;
    proxy_ssl_verify on;
    proxy_ssl_session_reuse on;
    proxy_set_header Connection '';
    proxy_set_header Host $s3_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Authorization '';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_buffering on;
    proxy_intercept_errors on;
    resolver 8.8.4.4 8.8.8.8;
    resolver_timeout 10s;
    proxy_pass https://$s3_host/$s3_bucket/$1;
}

Adding NGINX caching on-top of this is pretty trivial.

Also, heads up, in the directive proxycachepath, they should consider enabling "usetemppath". This directive instructs NGINX to write them to the same directories where they will be cached. We recommend that you set this parameter to off to avoid unnecessary copying of data between file systems. usetemppath was introduced in NGINX version 1.7.10 and NGINX Plus R6. 

use_temp_path=off

Also, they should enable "proxycacherevalidate". This saves on bandwidth, because the server sends the full item only if it has been modified since the time recorded in the Last-Modified header. 

proxy_cache_revalidate on;

Friday, March 10, 2017

learned a few new linux tricks:

my old colleague told me long time ago time is not the same as /usr/bin/time, there is a new article talks about this

I learned you can type \time (will search time in your $PATH) to not use the built-in time

the ‘ escape character just make it a different from time for bash, you can use t\ime` as well, same.

it's helpful that sometimes you don't want to use alias commands (like turn off output colors), you can use \ls to call /bin/ls instead of using your ls alias.

when you forgot use nohup for a long running command, you can ctrl-z to suspend it, bg it, and remember to disown it as well.

some examples here

molly-guard is a command prevants you accidentially reboot/shutdown your machine. check example of it

Calling Go Functions from Other Languages is an interesting read.

now I know I can call a go program in java, and of course, in luajit.

kotlin releases version 1.1, coroutines (async/await, yield) is nice.

borrowed Reactive Web Applications from local library.

actually I read the first chapter two years ago, but chapter 1 covers many history/concepts, I don't think I had the knowledge to understand them back then.

I really enjoy it and I need to read it again. (can download it free from official manning site)

rest of the book is about play and akka, I kinda interested in both but I don't want to touch scala, so I think it will be a very quick read.

recently I have the same problem as Wrangling State In Clojure

I used to use atom for common arguments shared between few functions. calling function is easier (fewer arguments), but quite difficult to test.

I switched to more explicit arguments passing, but it reminds me I need a deeper knowledge about Destructuring in Clojure

I read Clojure, The Good Parts again, this is gold:

Most of the time when using atoms, they should not be def at the top level. They should be returned from constructor functions, or stored in Component.

and the use of :aliases in project.clj (for lein):

{:aliases "build-foo" ["run" "-m" "rasterize.build.foo/build-foo"]}

finished reading Clojure Programming Cookbook, I learned quite a lot from it.

clojure really is a deep language, I kept learning new things from different people different books, it's lots of fun.

I also just tried Nightlight, by adding this to your profiles.clj:

{:user {:plugins [[nightlight/lein-nightlight "1.6.3"]]}}

you can have a web editor when you run lein nightlight

The System Design Primer provides some flash cards for AnkiDroid, quite useful.

Thursday, March 16, 2017

an good discussion about reading resources: Ask HN: What are some good technology blogs to follow?

found two excellent site from it:

also some other good readings:

Friday, March 17, 2017

using cider for quite a while, but didn't use its debugging function: cider/debugging

need spend some time on it.

Easily Deploy Your Clojure Web Site is the new ebook from author of Clojure for the Brave and True, free to read online.

also reading two books from leanpub:

Read Read-Eval-Print-λove by Michael Fogus, co-author of The Joy of Clojure, free to read online.

ClojureScript Unraveled, nice introduction to clojurescript.

listened to two cognicast:

Paul deGrandis - Cognicast Episode 118

talks about cognitect-labs/vase: Data driven microservices, which uses in-memory Datomic database by default.

and here is a tutorial for datomic: ftravers/datomic-tutorial

and Paul Stadig - Cognicast Episode 113

he wrote a book: Clojure Polymorphism

ctop - concise commandline monitoring for containers is a monitor command for containers, looks very nice.

another tool people recommend: Glances - An Eye on your system

couple more cool tools:

some other readings:

Tuesday, March 21, 2017

this is the most beautiful version of SICP I've seen: Structure and Interpretation of Computer Programs, 2e

saw Introduction to TLA+ by Leslie Lamport

don't know what TLA+ is, but SAM (State-Action-Model) pattern is built on TLA+, and it's for distributed system, worth taking a look.

pretty exciting news: A new way of blogging about Lua

it is using vvanders/wasm_lua: Lua VM running in a WASM environment

so javascript really becomes assembly language of web.

Wednesday, March 22, 2017

I wrote a disown note above (march 10), but I found another good one today from this thread: Famous ssh moment : ProgrammerHumor

ctrl+z
bg %%; disown %%

%% is the "current job" (last job stopped in foreground or started in background)

clojure destructuring, this signature can handle quite complex arguments:

(defn foo [x & {:keys [a b] :or {a 1, b 2}}] ...)

when coding with clojure, two things are really difficult: naming and api (function) design. need a lot of thinking to work out a clean one.

I'm reading The Joy of Clojure 2nd Edtion now, from cover to cover.

not to learn clojure but how to code clojure.

Thursday, March 23, 2017

few resource for naming things in clojure:

Saturday, March 25, 2017

I've been using more vim recently, learned something new:

cgn:

search for pattern, then cgn to replace pattern with text. way better than cw. gn visual selects the pattern, so if just want to remove text, use dgn.

good for bulk edit, can just hit n for next appearance and . to repeat the command.

pipe selected text to shell command

insert result from shell command is simple, just hit !! and enter the command.

when I write new puppet manifests, I will do !! and apt-cache something to grab package names, and then I will select and pipe them to awk to do some parsing:

hit ! after selected text, then enter something like awk -F, '{print $2}'

select result, hit :, then enter sort. this uses vim's built in sort function. to use external command, hit ! and run the command.

insert instead of replace

above piping will replace (filter, in vim's term)

I found that to insert / copy result only is not that simple, undo/redo is faster.

but if just preview, can use :w ! cmd.

visit man page with one key stroke

just hit K, opens man page of the program under cursor, super nice.

ZZ and ZQ

alternatives to :wq and :q!, I can hold shift and that's more lazy.

and lastly, open vim and run commands, they don't show up in .bash_history

but of course, emacs can do it easily as well, you just need more fingers:

new project needs laravel, "codeigniter is abandoned and shouldn't be use anymore, symfony is too complicated ...", and I need to learn laravel deeply.

I really hate those names, by reading their website you can see these Facades, Artisan, Illuminate, Eloquent, Homestead, Valet, Dusk, Envoy...

and they have Elixir and Mix names as well!! they both to compile assets, however, Mix is built on top of webpack, and the old verion, Elixir is on top of gulp...

ok, I'm not going to try to understand these.

codeigniter is really not perfect, but as least it makes sense and gets shit done.

Saturday, March 25, 2017

HelloFresh's Migration to a New API Gateway to Enable Microservices interests me because of some tools they picked (all open sourced):

even their api gateway is open sourced: hellofresh/janus

infoq has a series of articles on monolith:

super high quality, highly recommended.

for me, I think if you have different goup of engineers take care of each services, and have a centralized monitoring and building system, that's fine.

otherwise, a waste of efforts.

graalvm/truffleruby: A high performance implementation of the Ruby programming language. Built on the GraalVM by Oracle Labs.

TruffleRuby is a fork of JRuby, some people tested said it's fast.

found that I didn't have ruby setup for my emacs, so I started with:

first, install two gems, required by robe package below:

$ gem install pry pry-doc

then in emacs, install robe package, and add these lines (I use company):

(add-hook 'ruby-mode-hook 'robe-mode)
(eval-after-load 'company
  '(push 'company-robe company-backends))

to use it, m-x inf-ruby to start a irb repl, then m-x robe-start to connect to the it.

optional, install flymake-ruby package for syntax check, but I want to keep it lightweight, I didn't use it.

Sunday, March 26, 2017

learned a very useful trick from this two articles:

in a boot repl, you can add new dependencies whenever you want by 

(set-env! :dependencies '[[org.clojure/data.json "0.2.6"]])

and you can define a function for it:

(defn deps [new-deps]
  (merge-env! :dependencies new-deps))

simply has the sample effect of above line.

(deps '[[org.clojure/data.json "0.2.6"]])

add the deps function to build.boot file, and start repl with boot repl.

this is already very useful, but typing s-exp under repl still sucks ... inf-clojure to rescue:

install inf-clojure package in emacs, then add this line to init.el:

(setq inf-clojure-program "boot -C repl")

now m-x inf-clojure will start the boot repl, in any buffer after trigger m-x inf-clojure-minor-mode can send clojure codes to repl and even have auto-complete...

of course paredit is there as well, super nice.

Monday, March 27, 2017

When I search for difference between inf-clojure and cider, I found cider-scratch

m-x cider-scratch then m-x cider-jack-in, you can also have a quick repl without creating a new .clj file.

it is using boot already and loaded my ~/build.boot, I don't know why but it works out of the box, having the same functionality as inf-clojure.

Tuesday, March 28, 2017

Lumen is a micro-framework also by laravel, which is much better choice for simple apps.

it uses nikic/FastRoute for routing, which not the same as laravel, for example, optional parameters is done by []:

$app->get('/[{name}]', function ($name = null) use ($app) { ... }

views, this part is confusing, they said views is removed from v5.2.0, but added back in v5.2.1, latest documentation still doesn't have views section.

however, I tested views is working at v5.4.5:

create resources/views/foo.php, call in route callback:

return view('foo', ['name' => $name]);

I like plain php template rather than blade, so it's good enough for me.

the other feature they removed since v5.1 is sessions, by checking their old commits, you'll need these middlewares enabled in bootstrap/app.php:

$app->middleware([
    'Illuminate\Cookie\Middleware\EncryptCookies',
    'Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse',
    'Illuminate\Session\Middleware\StartSession',
]);

a simple way to have sessions in latest version is to use Symfony's HttpFoundation Component, it's alreay one of the dependencies of lumen, so no extra setups.

a quick test routes/web.php:

use Symfony\Component\HttpFoundation\Session\Session;

$app->get('/set', function() use ($app) {
  $s = new Session();
  $s->start();
  $s->set('foo', 'bar');
});

$app->get('/get', function() use ($app) {
  $s = new Session();
  $s->start();
  var_dump($s->get('foo'));
});

I've been thinking about this part as well. nowadays they do sessions in jwt and using state management tools like redux, it really not necessary to do server side sessions. I'm an old fashioned guy and I need my server side sessions.

finally for netbeans, there is barryvdh/laravel-ide-helper to generate autocompletion file for lumen

reading Professional Clojure, borrowed from local library.

first chapter is great. after coding clojure for a while, I agree it tells the right way to do clojure. simple is really not easy.

notes are taken here.

Thursday, March 30, 2017

Learn redis the hard way (in production) is a good read.

it have few real-world suggestions:

redis is (mostly) single threaded

however, as redis faq said, usually redis is either memory or network bound, very unlikely CPU becomes bottleneck.

also redis has background (mainly related to disk I/O) jobs assigned to other threads.

it also affects cpu monitoring in a multi-core instance, for example aws:

Redis: Since Redis is single-threaded, the threshold is calculated as (90 / number of processor cores). For example, suppose you are using a cache.m1.xlarge node, which has four cores. In this case, the threshold for CPUUtilization would be (90 / 4), or 22.5%.

BGSAVE

also mentioned here: Redis design patterns for high volume applications

for large datasets, may choose using cron to trigger BGSAVE.

multiple databases

antirez admits that multiple databases is a bad idea here

I understand how this can be useful, but unfortunately I consider Redis multiple database errors my worst decision in Redis design at all... without any kind of real gain, it makes the internals a lot more complex. The reality is that databases don't scale well for a number of reason, like active expire of keys and VM. If the DB selection can be performed with a string I can see this feature being used as a scalable O(1) dictionary layer, that instead it is not.

With DB numbers, with a default of a few DBs, we are communication better what this feature is and how can be used I think. I hope that at some point we can drop the multiple DBs support at all, but I think it is probably too late as there is a number of people relying on this feature for their work.

One instance per data context / feature

it's better to avoid one redis uses for all. different feature should have different redis instance with its own config.

connection pool

twitter/twemproxy (aka nutcracker) is a proxy for memcached and redis made by twitter few years ago.

under ubuntu 16.04, can install by apt-get install nutcracker

it's good practice that you have a proxy to handle connection back pressure.

some extended reading:

in order to handle data sets larger than memory, can setup Redis cluster for sharding.

note that you'll have at least 3 nodes to determinte majority of masters.

Redis Sentinel provides HA for redis. sentinel nodes should not be at same nodes as redis (more nodes!)

I wonder if sentinel only handles failover, can I use it as zookeeper? or use sentinel as a general purpose HA tool? (I don't have answer yet)

then I read this question: distributed - zookeeper vs redis server sync - Stack Overflow

learned that there're Physical clustering (pacemaker) and Logical clustering (curator, zookeeper)

continue reading professional clojure:

chapter 3

learned that I can replace http response :body with parsed data:

(update request :body cheshire.core/decode)

much cleaner than use let to bind the parsed result to a new var.

the storage backend may vary, the book uses defprotocol defines common methods and reify to create object implemented those methods.

I think that's neat.

I still sometimes confuse protocol and multimethods, got a clearer idea after read them couple more times:

defmulti defines dispatcher for different implementation based on incoming arguments and defmethod to implement the actual method.

a lot of tests examples in this chapter as well.

chapter 5

a nice way to build an hiccup element:

(into [:ul] (for ...))

Linkerd-tcp, a lightweight, service-discovery-aware, TLS-ing TCP load balancer.

linkerd is interesting piece of technology, a rpc proxy.

they have some nice concepts like: budget and deadline in retries

I'm waiting for their next article of A Service Mesh for Kubernetes series, which is Retry budgets, deadline propagation, and failing gracefully.

Software Engineering Daily has one episode about it: Scaling Twitter with Buoyant.io's William Morgan

another book from author of Clojure for the brave and true: Parallel Programming in Clojure with Reducers

not much people mentions reducer since transducer, it will be an interesting read.

there is an article about transducer though: Deep dive into a clojure transducer

even though I didn't touch common lisp anymore, I found Articulate Common Lisp nice, it teaches you how to write Common Lisp in 2017.

networknt/light-java: A fast, lightweight and more productive microservices framework

I saw this framework in a microservices benchmark result, this one is top of the chart with other frameworks written in go.

for vim, this is a must read: Vim Text Objects: The Definitive Guide

usually I'll use t or f (like dt"), from now will use text object.

Friday, March 31, 2017

Clojure/west 2017 videos (schedule) are releasing to ClojureTV channel

I just watched Core.Async in Use - Timothy Baldridge

the bad codes in it is just the way I write clojure! I still learning to think in a functional way.

some code snippets I took from the talk:

resize a batch:

(defn batch [in max-size]
  (let [out (chan 1)
        xf (comp cat (partition-all max-size))]
    (pipeline 1 out xf in)
    out))

or

(defn batch-xf [max-size]
  (comp cat (partition-all max-size)))

use transducer, the point here is the final fn to deal with final result. a single exit point.

(def xform (comp cat
                 (filter #(> % 10))
                 (take 10)))

(pages xform
       conj
       (fn [result]
         (println "Finished with " result)))

use Pedestal - Interceptors

it has enter, exit and on-error part, just like js callbacks, each function receives context ctx and do some mutation, like ring middlewares

(defrecord Interceptor [enter exit on-error])

(def pipeline [parse-params encode-body handler])

(defn parse-params-enter
  [{:keys [param-string] :as ctx}]
  (assoc ctx :params (split-params param-string)))

(defn encode-body-exit
  [{:keys [body] :as ctx}]
  (assoc ctx :body-string (json/encode body)))

(defn handler-enter
  [{:keys [params db-conn] :as ctx}]
  (go (assoc ctx :body (<! (get-record (:id params))))))

lastly, I think is the most useful one, dataflow (FRP):

you'll define inputs and outputs of each node, and then connect them together, like a graph structure

I like this pattern the most and will dig deeper into it.

(defn transducer-fn
  [xducer]
  (let [rf (fn [state msg]
             (emit state :out msg))]
    (xducer rf)))

(def filter-node {:inputs {:in (transducer-fn
                               (filter pos?))}
                  :outputs {:out []}})

also watching Faster Delivery with Pedestal and Vase - Paul deGrandis

I learned fnil function when browsing clojure documentations, I found it is useful.

today I read another article on nil: The case for and against Clojure predicates returning nil

and some real world experience of clojure: Fourteen Months with Clojure

listened to old The InfoQ Podcast archive, and found some great episodes:

Peter Bourgon on Gossip, Paxos, Microservices in Go, and CRDTs at SoundCloud

paxos is CP and gossip is AP

two CRDT system:

talks about go, Peter Bourgon has some good opnions on why go is good, especially for large teams. and good views on why go prefers no versioning.

Go kit - A toolkit for microservices

another good episode:

Chris Richardson on Domain-Driven Microservices Design

he answers a question bothers me for a long time: why CQRS?

the answer is in a distributed environment, you'll need a place to store and query stuffs, CQRS is the component to do this.

and read Domain-Driven Design, a book from 2003 to help you build better microservice architecture. it makes sense and much better than throw out some buzzwords.

some good articles on this site: White Papers | Redis Labs

Managing Transactions in Redise uses lua script to handle transaction.

in real world of course should do it in mysql or postgresql, but it's fun to try using lua script with redis

Blog Archive