Saturday, September 03, 2016
as a emacs user, I have big problem when Ctrl
key is not the most left-bottom key. (it's the fn
key on some laptops).
I tried switching Ctrl
and Caps Lock
before, didn't like it since I can reach my Ctrl
key using my palm.
need to try this trick again because of the keyboard layout.
for ubuntu, open gnome-tweak-tool
, under Typing
tab, Ctrl key position
and select Caps Lock as Ctrl
an interesting comparison on implementing websocket with different languages: Websocket Shootout: Clojure, C++, Elixir, Go, NodeJS, and Ruby
Elixir is cool, Clojure makes sense and atom
helps a lot.
since there're examples of using websockets, I tried a little bit myself to explore more about it.
websocket server using code snippet provided by the article above.
and a simple websocket client:
package main
import (
"fmt"
"log"
"golang.org/x/net/websocket"
)
var origin = "http://localhost/"
var url = "ws://localhost:3000/ws"
func main() {
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
message := []byte("{\"type\":\"echo\",\"payload\":10}")
_, err = ws.Write(message)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Send: %s\n", message)
var msg = make([]byte, 512)
_, err = ws.Read(msg)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Receive: %s\n", msg)
}
perhaps the best explaination on Consistent Hashing
Monday, September 05, 2016
a simple file server by go
package main
import "net/http"
func main() {
http.ListenAndServe(":3000", http.FileServer(http.Dir("/tmp")))
}
Tuesday, September 06, 2016
a funny program fro Go in Action book:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
var wg sync.WaitGroup
func init() {
rand.Seed(time.Now().UnixNano())
}
func main() {
court := make(chan int)
wg.Add(2)
go player("Nadal", court)
go player("Djokovic", court)
court <- 1
wg.Wait()
}
func player(name string, court chan int) {
defer wg.Done()
for {
ball, ok := <-court
if !ok {
fmt.Printf("Player %s Won\n", name)
return
}
n := rand.Intn(100)
if n%13 == 0 {
fmt.Printf("Player %s Missed\n", name)
close(court)
return
}
fmt.Printf("Player %s Hit %d\n", name, ball)
ball++
court <- ball
}
}
Wednesday, September 07, 2016
linux journal auguest issue has one article on libmill, including example of a tcp server.
reading Go in Practice, a more practical, cookbook style book.
chapter 2 already covers things I'm interested in: command line arguments, configuration files,
some suggestions for from the book:
- juju/gnuflag brings GNU-style(Linux) flags to Go.
- urfave/cli, cli framework for go.
configuration format covers json, yaml and ini, and mentioned coreos/etcd for distributed configuration store.
os.Getenv()
for configuration via environment variables.
handles http server gracefully shutdown by braintree/manners
Thursday, September 08, 2016
Go in Practice also mentioned grpc, a very simple example:
the protobuf:
syntax = "proto3";
package diu;
service Hello {
rpc Say (HelloRequest) returns (HelloResponse) {}
}
message HelloRequest {
string name = 1;
}
message HelloResponse {
string message = 1;
}
compile with:
protoc -I=. --go_out=plugins=grpc:. ./diu.proto
server:
package main
import (
pb "diu"
"golang.org/x/net/context"
"google.golang.org/grpc"
"net"
)
type server struct{}
func (s *server) Say(ctx context.Context, in *pb.HelloRequest) (*pb.HelloResponse, error) {
msg := "Hello " + in.Name + "!"
return &pb.HelloResponse{Message: msg}, nil
}
func main() {
l, _ := net.Listen("tcp", ":55555")
s := grpc.NewServer()
pb.RegisterHelloServer(s, &server{})
s.Serve(l)
}
client:
package main
import (
pb "diu"
"fmt"
"golang.org/x/net/context"
"google.golang.org/grpc"
)
func main() {
address := "localhost:55555"
conn, _ := grpc.Dial(address, grpc.WithInsecure())
defer conn.Close()
c := pb.NewHelloClient(conn)
name := "Inigo Montoya"
hr := &pb.HelloRequest{Name: name}
r, _ := c.Say(context.Background(), hr)
fmt.Println(r.Message)
}
actually same as the example in grpc package
Friday, September 09, 2016
want to try ssl termination using HAProxy, so spent some time on studying it:
sudo apt-get install haproxy-doc
then navigate to file:///usr/share/doc/haproxy-doc/html/intro.html
some notes were taken and saved to dev/haproxy
so a basic test (no ssl)...
multiple backend servers (change the port 3001
and response body):
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) {
for k, v := range req.Header {
fmt.Printf("%s - %s\n", k, v)
}
fmt.Fprint(res, "server one")
})
fmt.Println("listening on port 3001...")
http.ListenAndServe(":3001", nil)
}
just included the added part based on the default haproxy.cfg
:
defaults
# add X-Forwarded-For header to backend requests
option forwardfor
# reduces latency between HAProxy and client by closing connections but maintaining keep-alives.
option http-server-close
frontend www-http
bind 127.0.0.1:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
backend www-backend
server www-1 127.0.0.1:3001 check
server www-2 127.0.0.1:3002 check
server www-3 127.0.0.1:3003 check
server www-4 127.0.0.1:3004 check
server www-5 127.0.0.1:3005 check
sudo service restart haproxy
to restart and play with it.
the default logging is not verbose as I wanted, but need to check the manual for the setup.
when ps
, there're 3 haproxy processes are running, one of them is a wrapper
this thread may explain this situation, but need to find a more detailed explaination later.
sinfo
service is flooding my syslog
(ubuntu 16.04), to disable the log, add
StandardOutput=null
to [Service]
section of /lib/systemd/system/sinfo.service
then
$ sudo systemctl daemon-reload && sudo systemctl restart sinfo
when playing with cassandra, the cqlsh
is not working:
$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': TypeError('ref() does not take keyword arguments',)})
it's an error of incompatible python version, wtf, this is even worse than node.js ..
Saturday, September 10, 2016
I read many articles on mobile phone daily. I always want to have a simple way to send them to my kindle and read on it.
I couldn't find a good library for mobi conversion. so instead convert directly to mobi, what I need is:
first, a library that converts files to epub format: psiegman/epublib
then use amazon's kinglegen program to convert epub to mobi format.
emacs' artist mode is a useful tool when I need to mark down relationships between different components.
combine with ditaa and org-mode is even more powerful: What's New in Emacs 25.1, Ditaa and Artist-mode
a simple way to create a rum project:
lein new tenzing your-app +rum
tenzing is using boot
, so entering the new project and run boot dev
to start the dev server with auto reload.
Monday, September 12, 2016
midnight commander provides a wrapper shell script (/usr/lib/mc/mc-wrapper.sh
), use this wrapper if you want mc
exit to the directory you were in.
simply add alias mc='. /usr/lib/mc/mc-wrapper.sh'
to .bashrc
.
to copy full path of a file, use readlink
and xclip
:
readlink -fn . | xclip -sel c
~-
refers to the last directory you've visited ("$OLDPWD"
), so copy a file from the previous directory you visited:
cp ~-/foo.txt .
and ~+
refers to the current directory ("$PWD"
).
another way is use xclip-copyfile
/ xclip-pastefile
:
$ cd /tmp/
$ xclip-copyfile foo.txt
$ cd /opt/
$ xclip-pastefile
to copy a file from git history
$ git show dev-branch:root/src/foo.txt > foo.txt
$ git show rev:root/src/foo.txt > foo2.txt
to create a patch for the file:
$ git format-patch rev1..rev2 foo.txt
midnight commander has a patchfs virual file system, navigate to a .patch
file, hit enter
. then you can browse patch content like files. you can also edit or add/remove files. pretty cool.
midnight commander and emacs they both have a nice diff tool.
for mc
:
C-x C-d
to compare fileenter
to jump to next diff (p
to jump back)F5
to replace change with right,F15
to replace with left.F2
to save orF10
to quit
you can also start it with mcdiff file1 file2
ediff is more powerful, a simple apply changes from either file:
M-x ediff-files
, select files in mini buffern
/space
move to next diff,p
move to previousa
/b
to apply change fromA
/B
wa
to saveA
buffer,wb
to saveB
buffer,q
to exit
Tuesday, September 13, 2016
saw this quote from a hackernews thread: Golang concepts from an OOP point of view
"accept interfaces, return structs".
a good discussion on hackernews about REST anti-patterns,
more about HATEOAS (Hypermedia as the Engine of Application State):
re-using ip addresses in Vagrant vm causes ssh complaining, use these two config to remove those warnings.
Host 192.168.33.*
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
puppet is good at managing server states, but sometimes I want to run some one-time commands immediately. puppet can't do that.
MCollective is the puppet way to do it (they have another tool for enterprise: Application Orchestration )
mcollective is powerful, you can target group of servers using facter. however, setup mcollective requires activemq, another single point of failure. unless I have to manage hundred and thousand of servers, it doesn't make sense to add mcollective to the stack.
ansible is a reasonable tool for this kind of work, using purely ssh and yaml/ini. ansible doesn't manage states, so it only makes sense if you're building immutable instances.
many people use puppet and ansible together, so I guess the solution is either mcollective or setup a ansible inventory.
(puppet and chef are simuliar, salt also requires a zeromq)
I decided give ansible a try. a quick test for excuting a command on remote node:
vagrant up
few VMs, (use config.ssh.private_key_path
to specify the same private key)
setup the inventory by creating a hosts
file:
[target]
192.168.33.10
192.168.33.11
[target:vars]
ansible_user=vagrant
ansible_ssh_private_key_file=/path/to/vagrant/private_key
target
is the group name, use -i
to locate the hosts
file.
to list hosts:
ansible -i hosts target --list-hosts
perform a ping:
ansible -i hosts target -m ping
execute a command:
ansible -i hosts target -m shell -a "uptime"
x-tile is a useful tool to manage windows.
I will setup a custom layout, then x-tile 1
to tile windows, or x-tile w
brings up the main window for re-ordering. (Ctrl 1
to execute after re-ordered)
people said guix and nix are the future, made me really want to try GuixSD and NixOS.
I know I can use guix and nix on my ubuntu, but I think it's better to use the distribution.
nixos seems a bit more complete and popular at this moment, and using systemd.
but guix uses guile (the new elisp), which is a huge selling point.
I think I'll start with guixsd ...
here the installation guide: GNU Guix Reference Manual: USB Stick Installation
Wednesday, September 14, 2016
this article from facebook is very good: Scalable and secure access with SSH
- configured SSH servers to trust certificate authority (CA) and everything it signs.
- think of authentication and authorization separately. Authentication securely verifies that the client is who it claims to be but does not grant any permissions. After the successful authentication, authorization decides whether or not the client can perform a specific action.
- production systems only have local accounts in
/etc/passwd
. The most common account used isroot
. - CA itself determines who can use SSH and where.
- issue short-lived certificates.
- OpenSSH can give you detailed information on which certificate was used to authenticate. Even though the local account we use for login is root, for any other local account, the accounting system enables us to easily identify who logged in where.
- use a bastion host to reach production. Engineers can authenticate to those bastion hosts only from trusted networks.
- Once users are properly authenticated and authorized, a background process contacts our internal CA to request a signed SSH certificate. From the bastion host, having obtained the appropriate certificate, engineers can SSH into production machines as either root or other lower privileged users.
see the article for implementations.
netflix has a simuliar thing: bless: an SSH Certificate Authority that runs as a AWS Lambda function
another one is Gravitational Teleport
when installing guixsd, already learned couple things:
zile is a emacs-like editor, but much smaller. I can drop vim-tiny
from now.
a sample config file is included:
$ cp /usr/share/doc/zile/dotzile.sample ~/.zile
setup wireless network from cli:
save this to wpa_supplicant.conf
(can save to anywhere)
network={
ssid="my-ssid"
key_mgmt=WPA-PSK
psk="the network's secret passphrase"
}
then
$ wpa_supplicant -c wpa_supplicant.conf -i wlan0 -B
$ dhclient -v wlan0
Saturday, September 17, 2016
I was checking caching with nginx and memcached, found this ngxhttpenhancedmemcachedmodule, described in HTTP caching with Nginx and Memcached
then I check varnish for external storage, found out varnish 5 just released two days ago: Changes in Varnish 5.0
you can Separate VCL files for different kind of requests now.
of course people compare varnish with nginx: Boost Drupal 8 Performance with NGINX: Cache & Load Balance, but I think varnish's author also made good points when defending varnish.
for me, I like varnish just because nginx's if
is useless and doesn't make any sense.
and from a quote from discussion on hackernews
HAProxy is a better load balancer, Nginx is a better static content server, Pound is a better SSL terminator, Varnish is a better caching and compressing reverse proxy, etc.
I never heard Pound before, may worth check it out. (apt-get install pound
)
more on setup CA for ssh:
- How To Create an SSH CA to Validate Hosts and Clients with Ubuntu
- Using Teleport with OpenSSH - Gravitational Teleport
I never understand REST, so even I read There is No REST API, discussion on reddit, the thesis and the blog post, I still don't understand it.
Wednesday, September 21, 2016
found my ibook g4 when cleaning the house. It still has os x tiger running, but it's useless because all my development tools are not available anymore.
I tried installing linux before, didn't work out. I decided to try again. this time it works!
I saved the notes to the ibook g4 page.
Thursday, September 22, 2016
few notes from How Dropbox securely stores your passwords:
bcrypt
has length limitation, only 72 characters are used, so do a sha256
hash first.
for cost factor (10
in the article), see Recommended # of rounds for bcrypt
for the unique, per-user salt, as Secure Salted Password Hashing mentioned, Never reuse a salt.
mentioned in the article, using aes256
for global pepper) then you can rotate the key (a decrypt-and-encrypt step)
aes256(bcrypt(sha512(password), unique_per_user_salt, 10), global_pepper, unique_iv)
result and salt are saved in database and the global pepper is saved separately. iv must be unique, usually generated randomly.
playlist for Strange Loop 2016, don't have time to watch any of them yet.
there is one about clojure: "Agility & Robustness: Clojure spec" by Stuart Halloway
another conference I found is SREcon, here's the playlist for SREcon16 and SREcon15
I heard it from Building A Billion User Load Balancer
an interesting one: 5 Logstash Alternatives
because it covers rsyslog
, I prefer rsyslog
whenever possible for the reason that it comes with ubuntu.
their blog has lots of interesting things about rsyslog
some rsyslog modules they mentioned:
- imuxsock to get system logs
- imfile to tail files
- mmnormalize to parse
- omhiredis to support writing to redis
- omkafka to support writing to kafka
- omelasticsearch to support writing to elasticsearch
I also like the idea of using redis as buffer instead of kafka, I need to test it out.
a great place to discover fonts: Fonts ยท GitHub
many of them came from The League of Moveable Type
Friday, September 23, 2016
initialize a lein
project:
lein new usrcljs
I'm to tired of all the react wrapper/frameworks, I decided to play some pure clojurescript first.
a minimal project.clj
for clojurescript:
(defproject usrcljs "0.1.0-SNAPSHOT"
:plugins [[lein-figwheel "0.5.4-7"]
[lein-cljsbuild "1.1.4"]]
:dependencies [[org.clojure/clojure "1.8.0"]
[org.clojure/clojurescript "1.9.93"]]
:clean-targets [:target-path "out"]
:cljsbuild {:builds
{:dev
{:source-paths ["src"],
:figwheel true,
:compiler {:main "usrcljs.core"}}}})
figwheel
not only provides live codeing feature, but also a useful repl
which already connected to your page.
cljsbuild
also provides repl
, but it's not easy to setup a cljsbuild repl-launch
.
figwheel quick start contains a simple example for react as well.
more useful links:
- magomimmo/modern-cljs: A series of tutorials on ClojureScript
- ibdknox/jayq: A ClojureScript wrapper for jQuery
- r0man/sablono: Lisp/Hiccup style templating for Facebook's React in ClojureScript.
Monday, September 26, 2016
macbook pro keyboard is not friendly to emacs user, the following will map command
key as meta
(alt
):
(when (eq system-type 'darwin)
(setq mac-option-key-is-meta nil
mac-command-key-is-meta t
mac-command-modifier 'meta
mac-option-modifier 'none))
os x can change modifiers from settings (like change caps locks
to ctrl
), unless you need to separate left and right keys, which you'll need something like Karabiner.
a re-post article My love-hate relationship with LuaJIT on hackernews caught my attention.
TIL LuaJIT is not Lua, and LuaJIT's author Mike Pall
I'm not a C user, but quite interested in LuaJIT, it's so tiny!
tig is such a nice tool. I just use it to browser repository, it's very fast and easy:
| m | Switch to main view. |
| d | Switch to diff view. |
| l | Switch to log view. |
| t | Switch to (directory) tree view. |
| g | Switch to grep view. |
| b | Switch to blame view. |
| r | Switch to refs view. |
| y | Switch to stash view. |
that's pretty much the keys I needed.
I'll work on some CodeIgniter projects, which I never touched before.
by reading the documentations, I believe symfony is a better and mature framework than CI.
I'll drop some notes on my CodeIgniter page.
Tuesday, September 27, 2016
emacs --daemon
always quits under os x, so I changed to (server-start)
and lock one buffer to prevent emacs exits.
(defun usrj/lock-scratch-buffer ()
(set-buffer "*scratch*")
(emacs-lock-mode))
(add-hook 'after-init-hook 'usrj/lock-scratch-buffer t)
(require 'server)
(unless (server-running-p)
(server-start))
to always keep buffer in a window, use set-window-dedicated-p
and (setq pop-up-windows -1)
snippet from here:
(defun toggle-window-dedicated ()
"Control whether or not Emacs is allowed to display another
buffer in current window."
(interactive)
(message
(if (let (window (get-buffer-window (current-buffer)))
(set-window-dedicated-p window (not (window-dedicated-p window))))
"%s: Can't touch this!"
"%s is up for grabs.")
(current-buffer)))
(global-set-key (kbd "C-c t") 'toggle-window-dedicated)
(update, new frames kept popping up once I dedicated a window, I changed to use winner-mode instead)
my laptop is pretty old and doesn't work very well with vagrant, I'll move my development vm to digitalocean, use owainlewis/digital-ocean to manage.
switched back to gnome-flashback
from xfce
.
xfce
is quite buggy, like missing icons, font size changes every time I logged in. and movement of windows/mouse not as smooth as gnome-flashback
.
I just feel more productive under gnome-flashback
.
Wednesday, September 28, 2016
noplay/python-mysql-replication: Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL is an interesting project, I'll take a look at the libraries they used and see if I could find anything useful for me.
about clojure and emacs: Emacs is hurting Clojure
I started with emacs because of common lisp, and use it for clojure naturally. but I agree if I started with both of them at the beginning, I probably gave up already.
Cursive is nice, I tried but returned to emacs just because idea is more heavy. since I'll code kotlin occasionally using idea, I should give another try.
two apps built with clojurescript
when I choosing a frontend clojurescript library, I checked with re-frame, but it's too complicated and solve complicated UI problems.
it's nice that yogthos posts an example on reddit: Reusable Components
I found myself don't really know defrecord
and didn't use it that much. I should re-read Creating and Extending Abstractions with Multimethods, Protocols, and Records and Datatypes: deftype, defrecord and reify
finally got a chance to test using certificate to login ssh server, followed this article: Scalable and secure access with SSH
you have 3 machines:
C
: ca server (to issue signed ca)S
: server (production server in the article)U
: user machine (bastion host in the article)
in C
, create ca pair:
$ ssh-keygen -C CA -f ca
(you got ca
and ca.pub
), copy ca.pub
to S:/etc/ssh/ca.pub
)
in S
, add to /etc/ssh/sshd_config
:
TrustedUserCAKeys /etc/ssh/ca.pub
then restart ssh server.
in U
, generate key pair:
$ ssh-keygen -t ecdsa
(you got id_ecdsa
and id_ecdsa.pub
), copy id_ecdsa.pub
to C
in C
, sign it (man ssh-keygen
to see meaning of each option):
$ ssh-keygen -s ca -I jim -n root -V +1w -z 1 id_ecdsa.pub
(you got id_ecdsa-cert.pub
), copy id_ecdsa-cert.pub
back to U
in U
, to inspect:
$ ssh-keygen -Lf id_ecdsa-cert.pub
ssh to S
with
$ ssh -i id_ecdsa root@S
you should be able to see logs in /var/log/auth.log
to support principals, in S
:
add to /etc/ssh/sshd_config
:
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
then
$ mkdir /etc/ssh/auth_principals
$ echo -e 'zone-webservers\nroot-everywhere' > /etc/ssh/auth_principals/root
back to U
, when you tried ssh again, it failed and there're error messages in S:/var/log/auth.log
.
we need to re-sign the key with principal:
in C
, sign id_ecdsa.pub
again with:
$ ssh-keygen -s ca -I jim -n zone-webservers -V +1w -z 2 id_ecdsa.pub
(you got id_ecdsa-cert.pub
again, but -z 2
is able to distinct the new key with the previous one)
copy id_ecdsa-cert.pub
back to U
.
in U
, ssh to S
again, this time should be success.
reading Programming in Lua, lua is a simple language, like a combination of php and go. I don't have problem with it so far.
I'll upload my notes to read/programming-lua.
Thursday, September 29, 2016
GOTO Conferences Chicago 2016 playlist, so many good topics.
Toronto Java Users Group auguest topic: The JVM Does What?!
Blog Archive
- Newer Entries
- 2016 October
- 2016 November
- 2016 December
- 2017 January
- 2017 February
- 2017 March
- 2017 April
- 2017 May
- 2017 June
- 2017 July
- 2017 August
- 2017 September
- 2017 October
- 2017 November
- 2017 December
- 2018 January
- 2018 February
- 2018 March
- 2018 April
- 2018 May
- 2018 June
- 2018 July
- 2018 August
- 2018 September
- 2018 October
- 2018 November
- 2018 December
- 2019 January
- 2019 February
- 2019 March
- 2019 April
- 2019 May
- 2019 July
- 2019 October
- 2019 November
- 2019 December
- 2020 August
- 2020 September
- 2020 October
- 2020 November
- 2020 December
- 2021 January
- 2021 February
- 2021 March
- 2021 April
- 2021 May
- 2021 June
- 2021 August
- 2021 September
- 2021 December
- 2022 March
- 2022 April
- 2022 May
- 2022 June
- 2022 July
- 2022 August
- 2022 September
- 2022 October
- 2022 November
- 2022 December
- 2023 January
- 2023 February
- 2023 March
- 2023 April
- 2023 July
- 2023 August
- 2023 September
- 2023 October
- 2023 November
- 2023 December
- 2024 January
- 2024 February
- 2024 March
- 2024 April
- 2024 May
- 2024 June
- 2024 August
- 2024 September
- Older Entries
- 2016 August
- 2016 July
- 2016 June
- 2016 May
- 2016 April
- 2016 March
- 2016 February
- 2016 January
- 2015 December
- 2015 November
- 2015 October
- 2015 September
- 2015 August
- 2015 July
- 2015 June
- 2015 May
- 2015 April
- 2015 March
- 2015 February
- 2015 January
- 2014 December
- 2014 November
- 2014 October
- 2014 September
- 2014 August
- 2014 March
- 2014 February
- 2014 January
- 2013 December
- 2013 October
- 2013 July
- 2013 June
- 2013 May
- 2013 March
- 2013 February
- 2013 January
- 2012 December
- 2012 November
- 2012 October
- 2012 September
- 2012 August