jchk - /blog/2016-09

Saturday, September 03, 2016

as a emacs user, I have big problem when Ctrl key is not the most left-bottom key. (it's the fn key on some laptops).

I tried switching Ctrl and Caps Lock before, didn't like it since I can reach my Ctrl key using my palm.

need to try this trick again because of the keyboard layout.

for ubuntu, open gnome-tweak-tool, under Typing tab, Ctrl key position and select Caps Lock as Ctrl

an interesting comparison on implementing websocket with different languages: Websocket Shootout: Clojure, C++, Elixir, Go, NodeJS, and Ruby

Elixir is cool, Clojure makes sense and atom helps a lot.

since there're examples of using websockets, I tried a little bit myself to explore more about it.

websocket server using code snippet provided by the article above.

and a simple websocket client:

package main

import (
	"fmt"
	"log"

	"golang.org/x/net/websocket"
)

var origin = "http://localhost/"
var url = "ws://localhost:3000/ws"

func main() {
	ws, err := websocket.Dial(url, "", origin)
	if err != nil {
		log.Fatal(err)
	}

	message := []byte("{\"type\":\"echo\",\"payload\":10}")
	_, err = ws.Write(message)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Send: %s\n", message)

	var msg = make([]byte, 512)
	_, err = ws.Read(msg)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Receive: %s\n", msg)
}

perhaps the best explaination on Consistent Hashing

Monday, September 05, 2016

a simple file server by go

package main

import "net/http"

func main() {
	http.ListenAndServe(":3000", http.FileServer(http.Dir("/tmp")))
}

Tuesday, September 06, 2016

a funny program fro Go in Action book:

package main

import (
	"fmt"
	"math/rand"
	"sync"
	"time"
)

var wg sync.WaitGroup

func init() {
	rand.Seed(time.Now().UnixNano())
}

func main() {
	court := make(chan int)

	wg.Add(2)

	go player("Nadal", court)
	go player("Djokovic", court)

	court <- 1

	wg.Wait()
}

func player(name string, court chan int) {
	defer wg.Done()

	for {
		ball, ok := <-court
		if !ok {
			fmt.Printf("Player %s Won\n", name)
			return
		}

		n := rand.Intn(100)
		if n%13 == 0 {
			fmt.Printf("Player %s Missed\n", name)
			close(court)
			return
		}

		fmt.Printf("Player %s Hit %d\n", name, ball)
		ball++

		court <- ball
	}
}

Wednesday, September 07, 2016

linux journal auguest issue has one article on libmill, including example of a tcp server.

reading Go in Practice, a more practical, cookbook style book.

chapter 2 already covers things I'm interested in: command line arguments, configuration files,

some suggestions for from the book:

juju/gnuflag brings GNU-style(Linux) flags to Go.
urfave/cli, cli framework for go.

configuration format covers json, yaml and ini, and mentioned coreos/etcd for distributed configuration store.

os.Getenv() for configuration via environment variables.

handles http server gracefully shutdown by braintree/manners

Thursday, September 08, 2016

Go in Practice also mentioned grpc, a very simple example:

the protobuf:

syntax = "proto3";
package diu;
service Hello {
    rpc Say (HelloRequest) returns (HelloResponse) {}
}
message HelloRequest {
    string name = 1;
}
message HelloResponse {
    string message = 1;
}

compile with:

protoc -I=. --go_out=plugins=grpc:. ./diu.proto

server:

package main

import (
	pb "diu"
	"golang.org/x/net/context"
	"google.golang.org/grpc"
	"net"
)

type server struct{}

func (s *server) Say(ctx context.Context, in *pb.HelloRequest) (*pb.HelloResponse, error) {
	msg := "Hello " + in.Name + "!"
	return &pb.HelloResponse{Message: msg}, nil
}
func main() {
	l, _ := net.Listen("tcp", ":55555")
	s := grpc.NewServer()
	pb.RegisterHelloServer(s, &server{})
	s.Serve(l)
}

client:

package main

import (
	pb "diu"
	"fmt"
	"golang.org/x/net/context"
	"google.golang.org/grpc"
)

func main() {
	address := "localhost:55555"
	conn, _ := grpc.Dial(address, grpc.WithInsecure())
	defer conn.Close()
	c := pb.NewHelloClient(conn)
	name := "Inigo Montoya"
	hr := &pb.HelloRequest{Name: name}
	r, _ := c.Say(context.Background(), hr)
	fmt.Println(r.Message)
}

actually same as the example in grpc package

Friday, September 09, 2016

want to try ssl termination using HAProxy, so spent some time on studying it:

sudo apt-get install haproxy-doc

then navigate to file:///usr/share/doc/haproxy-doc/html/intro.html

some notes were taken and saved to dev/haproxy

so a basic test (no ssl)...

multiple backend servers (change the port 3001 and response body):

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) {
		for k, v := range req.Header {
			fmt.Printf("%s - %s\n", k, v)
		}
		fmt.Fprint(res, "server one")
	})
	fmt.Println("listening on port 3001...")
	http.ListenAndServe(":3001", nil)
}

just included the added part based on the default haproxy.cfg:

defaults
    # add X-Forwarded-For header to backend requests
    option forwardfor
    
    # reduces latency between HAProxy and client by closing connections but maintaining keep-alives.
    option http-server-close

frontend www-http
    bind 127.0.0.1:80
    reqadd X-Forwarded-Proto:\ http
    default_backend www-backend

backend www-backend
    server www-1 127.0.0.1:3001 check
    server www-2 127.0.0.1:3002 check
    server www-3 127.0.0.1:3003 check
    server www-4 127.0.0.1:3004 check
    server www-5 127.0.0.1:3005 check

sudo service restart haproxy to restart and play with it.

the default logging is not verbose as I wanted, but need to check the manual for the setup.

when ps, there're 3 haproxy processes are running, one of them is a wrapper

this thread may explain this situation, but need to find a more detailed explaination later.

sinfo service is flooding my syslog (ubuntu 16.04), to disable the log, add

StandardOutput=null

to [Service] section of /lib/systemd/system/sinfo.service

then

$ sudo systemctl daemon-reload && sudo systemctl restart sinfo

when playing with cassandra, the cqlsh is not working:

$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': TypeError('ref() does not take keyword arguments',)})

it's an error of incompatible python version, wtf, this is even worse than node.js ..

Saturday, September 10, 2016

I read many articles on mobile phone daily. I always want to have a simple way to send them to my kindle and read on it.

I couldn't find a good library for mobi conversion. so instead convert directly to mobi, what I need is:

first, a library that converts files to epub format: psiegman/epublib

then use amazon's kinglegen program to convert epub to mobi format.

emacs' artist mode is a useful tool when I need to mark down relationships between different components.

combine with ditaa and org-mode is even more powerful: What's New in Emacs 25.1, Ditaa and Artist-mode

a simple way to create a rum project:

lein new tenzing your-app +rum

tenzing is using boot, so entering the new project and run boot dev to start the dev server with auto reload.

Monday, September 12, 2016

midnight commander provides a wrapper shell script (/usr/lib/mc/mc-wrapper.sh), use this wrapper if you want mc exit to the directory you were in.

simply add alias mc='. /usr/lib/mc/mc-wrapper.sh' to .bashrc.

to copy full path of a file, use readlink and xclip:

readlink -fn . | xclip -sel c

~- refers to the last directory you've visited ("$OLDPWD"), so copy a file from the previous directory you visited:

cp ~-/foo.txt .

and ~+ refers to the current directory ("$PWD").

another way is use xclip-copyfile / xclip-pastefile:

$ cd /tmp/
$ xclip-copyfile foo.txt
$ cd /opt/
$ xclip-pastefile

to copy a file from git history

$ git show dev-branch:root/src/foo.txt > foo.txt

$ git show rev:root/src/foo.txt > foo2.txt

to create a patch for the file:

$ git format-patch rev1..rev2 foo.txt

midnight commander has a patchfs virual file system, navigate to a .patch file, hit enter. then you can browse patch content like files. you can also edit or add/remove files. pretty cool.

midnight commander and emacs they both have a nice diff tool.

for mc:

C-x C-d to compare file
enter to jump to next diff (p to jump back)
F5 to replace change with right, F15 to replace with left.
F2 to save or F10 to quit

you can also start it with mcdiff file1 file2

ediff is more powerful, a simple apply changes from either file:

M-x ediff-files, select files in mini buffer
n / space move to next diff, p move to previous
a / b to apply change from A / B
wa to save A buffer, wb to save B buffer, q to exit

Tuesday, September 13, 2016

saw this quote from a hackernews thread: Golang concepts from an OOP point of view

"accept interfaces, return structs".

a good discussion on hackernews about REST anti-patterns,

more about HATEOAS (Hypermedia as the Engine of Application State):

re-using ip addresses in Vagrant vm causes ssh complaining, use these two config to remove those warnings.

Host 192.168.33.*
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null

puppet is good at managing server states, but sometimes I want to run some one-time commands immediately. puppet can't do that.

MCollective is the puppet way to do it (they have another tool for enterprise: Application Orchestration )

mcollective is powerful, you can target group of servers using facter. however, setup mcollective requires activemq, another single point of failure. unless I have to manage hundred and thousand of servers, it doesn't make sense to add mcollective to the stack.

ansible is a reasonable tool for this kind of work, using purely ssh and yaml/ini. ansible doesn't manage states, so it only makes sense if you're building immutable instances.

many people use puppet and ansible together, so I guess the solution is either mcollective or setup a ansible inventory.

(puppet and chef are simuliar, salt also requires a zeromq)

I decided give ansible a try. a quick test for excuting a command on remote node:

vagrant up few VMs, (use config.ssh.private_key_path to specify the same private key)

setup the inventory by creating a hosts file:

[target]
192.168.33.10
192.168.33.11

[target:vars]
ansible_user=vagrant
ansible_ssh_private_key_file=/path/to/vagrant/private_key

target is the group name, use -i to locate the hosts file.

to list hosts:

ansible -i hosts target --list-hosts

perform a ping:

ansible -i hosts target -m ping

execute a command:

ansible -i hosts target -m shell -a "uptime"

x-tile is a useful tool to manage windows.

I will setup a custom layout, then x-tile 1 to tile windows, or x-tile w brings up the main window for re-ordering. (Ctrl 1 to execute after re-ordered)

people said guix and nix are the future, made me really want to try GuixSD and NixOS.

I know I can use guix and nix on my ubuntu, but I think it's better to use the distribution.

nixos seems a bit more complete and popular at this moment, and using systemd.

but guix uses guile (the new elisp), which is a huge selling point.

I think I'll start with guixsd ...

here the installation guide: GNU Guix Reference Manual: USB Stick Installation

Wednesday, September 14, 2016

this article from facebook is very good: Scalable and secure access with SSH

configured SSH servers to trust certificate authority (CA) and everything it signs.
think of authentication and authorization separately. Authentication securely verifies that the client is who it claims to be but does not grant any permissions. After the successful authentication, authorization decides whether or not the client can perform a specific action.
production systems only have local accounts in /etc/passwd. The most common account used is root.
CA itself determines who can use SSH and where.
issue short-lived certificates.
OpenSSH can give you detailed information on which certificate was used to authenticate. Even though the local account we use for login is root, for any other local account, the accounting system enables us to easily identify who logged in where.
use a bastion host to reach production. Engineers can authenticate to those bastion hosts only from trusted networks.
Once users are properly authenticated and authorized, a background process contacts our internal CA to request a signed SSH certificate. From the bastion host, having obtained the appropriate certificate, engineers can SSH into production machines as either root or other lower privileged users.

see the article for implementations.

netflix has a simuliar thing: bless: an SSH Certificate Authority that runs as a AWS Lambda function

another one is Gravitational Teleport

when installing guixsd, already learned couple things:

zile is a emacs-like editor, but much smaller. I can drop vim-tiny from now.

a sample config file is included:

$ cp /usr/share/doc/zile/dotzile.sample ~/.zile

setup wireless network from cli:

save this to wpa_supplicant.conf (can save to anywhere)

network={
  ssid="my-ssid"
  key_mgmt=WPA-PSK
  psk="the network's secret passphrase"
}

then

$ wpa_supplicant -c wpa_supplicant.conf -i wlan0 -B
$ dhclient -v wlan0

Saturday, September 17, 2016

I was checking caching with nginx and memcached, found this ngxhttpenhancedmemcachedmodule, described in HTTP caching with Nginx and Memcached

then I check varnish for external storage, found out varnish 5 just released two days ago: Changes in Varnish 5.0

you can Separate VCL files for different kind of requests now.

of course people compare varnish with nginx: Boost Drupal 8 Performance with NGINX: Cache & Load Balance, but I think varnish's author also made good points when defending varnish.

for me, I like varnish just because nginx's if is useless and doesn't make any sense.

and from a quote from discussion on hackernews

HAProxy is a better load balancer, Nginx is a better static content server, Pound is a better SSL terminator, Varnish is a better caching and compressing reverse proxy, etc.

I never heard Pound before, may worth check it out. (apt-get install pound)

Wednesday, September 21, 2016

found my ibook g4 when cleaning the house. It still has os x tiger running, but it's useless because all my development tools are not available anymore.

I tried installing linux before, didn't work out. I decided to try again. this time it works!

I saved the notes to the ibook g4 page.

Thursday, September 22, 2016

few notes from How Dropbox securely stores your passwords:

bcrypt has length limitation, only 72 characters are used, so do a sha256 hash first.

for cost factor (10 in the article), see Recommended # of rounds for bcrypt

for the unique, per-user salt, as Secure Salted Password Hashing mentioned, Never reuse a salt.

mentioned in the article, using aes256 for global pepper) then you can rotate the key (a decrypt-and-encrypt step)

aes256(bcrypt(sha512(password), unique_per_user_salt, 10), global_pepper, unique_iv)

result and salt are saved in database and the global pepper is saved separately. iv must be unique, usually generated randomly.

playlist for Strange Loop 2016, don't have time to watch any of them yet.

there is one about clojure: "Agility & Robustness: Clojure spec" by Stuart Halloway

another conference I found is SREcon, here's the playlist for SREcon16 and SREcon15

I heard it from Building A Billion User Load Balancer

an interesting one: 5 Logstash Alternatives

because it covers rsyslog, I prefer rsyslog whenever possible for the reason that it comes with ubuntu.

their blog has lots of interesting things about rsyslog

some rsyslog modules they mentioned:

imuxsock to get system logs
imfile to tail files
mmnormalize to parse
omhiredis to support writing to redis
omkafka to support writing to kafka
omelasticsearch to support writing to elasticsearch

I also like the idea of using redis as buffer instead of kafka, I need to test it out.

a great place to discover fonts: Fonts · GitHub

many of them came from The League of Moveable Type

Friday, September 23, 2016

initialize a lein project:

lein new usrcljs

I'm to tired of all the react wrapper/frameworks, I decided to play some pure clojurescript first.

a minimal project.clj for clojurescript:

(defproject usrcljs "0.1.0-SNAPSHOT"
  :plugins [[lein-figwheel "0.5.4-7"]
            [lein-cljsbuild "1.1.4"]]
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [org.clojure/clojurescript "1.9.93"]]
  :clean-targets [:target-path "out"]
  :cljsbuild {:builds
              {:dev
               {:source-paths ["src"],
                :figwheel true,
                :compiler {:main "usrcljs.core"}}}})

figwheel not only provides live codeing feature, but also a useful repl which already connected to your page.

cljsbuild also provides repl, but it's not easy to setup a cljsbuild repl-launch.

figwheel quick start contains a simple example for react as well.

Monday, September 26, 2016

macbook pro keyboard is not friendly to emacs user, the following will map command key as meta (alt):

(when (eq system-type 'darwin)
  (setq mac-option-key-is-meta nil
        mac-command-key-is-meta t
        mac-command-modifier 'meta
        mac-option-modifier 'none))

os x can change modifiers from settings (like change caps locks to ctrl), unless you need to separate left and right keys, which you'll need something like Karabiner.

a re-post article My love-hate relationship with LuaJIT on hackernews caught my attention.

TIL LuaJIT is not Lua, and LuaJIT's author Mike Pall

I'm not a C user, but quite interested in LuaJIT, it's so tiny!

tig is such a nice tool. I just use it to browser repository, it's very fast and easy:

| m | Switch to main view.             |
| d | Switch to diff view.             |
| l | Switch to log view.              |
| t | Switch to (directory) tree view. |
| g | Switch to grep view.             |
| b | Switch to blame view.            |
| r | Switch to refs view.             |
| y | Switch to stash view.            |

that's pretty much the keys I needed.

I'll work on some CodeIgniter projects, which I never touched before.

by reading the documentations, I believe symfony is a better and mature framework than CI.

I'll drop some notes on my CodeIgniter page.

Tuesday, September 27, 2016

emacs --daemon always quits under os x, so I changed to (server-start) and lock one buffer to prevent emacs exits.

(defun usrj/lock-scratch-buffer ()
  (set-buffer "*scratch*")
  (emacs-lock-mode))

(add-hook 'after-init-hook 'usrj/lock-scratch-buffer t)

(require 'server)
(unless (server-running-p)
  (server-start))

to always keep buffer in a window, use set-window-dedicated-p and (setq pop-up-windows -1)

snippet from here:

(defun toggle-window-dedicated ()
  "Control whether or not Emacs is allowed to display another
buffer in current window."
  (interactive)
  (message
   (if (let (window (get-buffer-window (current-buffer)))
         (set-window-dedicated-p window (not (window-dedicated-p window))))
       "%s: Can't touch this!"
     "%s is up for grabs.")
   (current-buffer)))

(global-set-key (kbd "C-c t") 'toggle-window-dedicated)

(update, new frames kept popping up once I dedicated a window, I changed to use winner-mode instead)

my laptop is pretty old and doesn't work very well with vagrant, I'll move my development vm to digitalocean, use owainlewis/digital-ocean to manage.

switched back to gnome-flashback from xfce.

xfce is quite buggy, like missing icons, font size changes every time I logged in. and movement of windows/mouse not as smooth as gnome-flashback.

I just feel more productive under gnome-flashback.

Wednesday, September 28, 2016

noplay/python-mysql-replication: Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL is an interesting project, I'll take a look at the libraries they used and see if I could find anything useful for me.

about clojure and emacs: Emacs is hurting Clojure

I started with emacs because of common lisp, and use it for clojure naturally. but I agree if I started with both of them at the beginning, I probably gave up already.

Cursive is nice, I tried but returned to emacs just because idea is more heavy. since I'll code kotlin occasionally using idea, I should give another try.

two apps built with clojurescript

when I choosing a frontend clojurescript library, I checked with re-frame, but it's too complicated and solve complicated UI problems.

it's nice that yogthos posts an example on reddit: Reusable Components

I found myself don't really know defrecord and didn't use it that much. I should re-read Creating and Extending Abstractions with Multimethods, Protocols, and Records and Datatypes: deftype, defrecord and reify

finally got a chance to test using certificate to login ssh server, followed this article: Scalable and secure access with SSH

you have 3 machines:

C: ca server (to issue signed ca)
S: server (production server in the article)
U: user machine (bastion host in the article)

in C, create ca pair:

$ ssh-keygen -C CA -f ca

(you got ca and ca.pub), copy ca.pub to S:/etc/ssh/ca.pub)

in S, add to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/ca.pub

then restart ssh server.

in U, generate key pair:

$ ssh-keygen -t ecdsa

(you got id_ecdsa and id_ecdsa.pub), copy id_ecdsa.pub to C

in C, sign it (man ssh-keygen to see meaning of each option):

$ ssh-keygen -s ca -I jim -n root -V +1w -z 1 id_ecdsa.pub

(you got id_ecdsa-cert.pub), copy id_ecdsa-cert.pub back to U

in U, to inspect:

$ ssh-keygen -Lf id_ecdsa-cert.pub

ssh to S with

$ ssh -i id_ecdsa root@S

you should be able to see logs in /var/log/auth.log

to support principals, in S:

add to /etc/ssh/sshd_config:

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

then

$ mkdir /etc/ssh/auth_principals
$ echo -e 'zone-webservers\nroot-everywhere' > /etc/ssh/auth_principals/root

back to U, when you tried ssh again, it failed and there're error messages in S:/var/log/auth.log.

we need to re-sign the key with principal:

in C, sign id_ecdsa.pub again with:

$ ssh-keygen -s ca -I jim -n zone-webservers -V +1w -z 2 id_ecdsa.pub

(you got id_ecdsa-cert.pub again, but -z 2 is able to distinct the new key with the previous one)

copy id_ecdsa-cert.pub back to U.

in U, ssh to S again, this time should be success.

reading Programming in Lua, lua is a simple language, like a combination of php and go. I don't have problem with it so far.

I'll upload my notes to read/programming-lua.

Thursday, September 29, 2016

GOTO Conferences Chicago 2016 playlist, so many good topics.

Toronto Java Users Group auguest topic: The JVM Does What?!

Blog Archive