Jim Cheung

Thursday, January 05, 2017

I'm looking for a monitoring tool again, here's a short list of candidates:

Icinga 2 actually quite good. the setup is not as difficult as Bosun.

they actually have a vagrant box for quick test: icinga-vagrant

their config file syntax makes sense to me, readable and easy to maintain.

I also like icingaweb2, can pick between graphite and pnp for graphs.

they have an online registry: Icinga Exchange, compatible with Monitoring Plugins, able to send notification to slack.

also have two essential features: API and CLI,

couple interesting article on clojure:

I spent less time on programming last couple weeks:

bought a SFC30 GamePad, I used it to play sfc games on my phone.

learning to play Rubik's Cube.

bought a Objective2 Headphone Amplifier from Mayflower Electronics, it's powerful enough to push AKG K612 Pro.

Wednesday, January 11, 2017

icinga2 is nice, but still too difficult to setup with graphs.

I then tried munin, setup is very simple, for both master and node.

munin is good for historical graph using rrdtool. however, its interval is 5 minutes, you must have something like icinga2 for real time monitoring (alerts), and use munin for historical graphs.

collectd is also very easy to setup, but I didn't find a good frontend for it yet.

I may try Ganglia when I have time, I read the book before, but didn't pay much attention to it. ganglia seems fit my only requirement: simple setup.

simple guide to setup munin on ubuntu (all run as root):

on master:

$ apt-get install munin munin-node spawn-fcgi

start cgi process:

$ spawn-fcgi -s /var/run/munin/fcgi-graph.sock -U www-data -u www-data -g www-data /usr/lib/munin/cgi/munin-cgi-graph

(when you find connection refuse error, try run the munin-cgi-graph in console, it just a perl script and will tell you which perl modules are missing.)

add to nginx:

location /munin/static/ {
  alias /etc/munin/static/;
  expires modified +1w;
}

location /munin/ {
  auth_basic            "Restricted";
  auth_basic_user_file  /etc/nginx/htpasswd;

  alias /var/cache/munin/www/;
  expires modified +310s;
}

location ^~ /munin-cgi/munin-cgi-graph/ {
  access_log off;
  fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*);
  fastcgi_param PATH_INFO $fastcgi_path_info;
  fastcgi_pass unix:/var/run/munin/fcgi-graph.sock;
  include fastcgi_params;
}

config munin-node:

$ munin-node-configure

this list all plugins. add --suggest it will display more useful information:

$ munin-node-configure --suggest

once you fixed the problem listed, can run

$ munin-node-configure --shell

it shows all available soft link commands.

for example, to enable mysql plugins, first update /etc/munin/plugin-conf.d/munin-node

check with:

$ munin-node-configure --suggest

activate the plugin:

$ ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins

then restart munin-node service:

$ service munin-node restart

test with stress:

$ stress --cpu 20 --timeout 180

for git. sometimes when I need to commit but want to skip one file, can try

$ git add -u

this will stage your changes, then unstage the file you want to skip:

$ git reset -- somefile

after this, git diff won't show you changes between uncommitted and head. use this instead: 

git diff --staged

also, don't commit with git commit -am, -a commits all changes, including the file you want to skip.

Wednesday, January 11, 2017

love munin, very easy to write plugins

there's alternative frontend: bootstrap 3 template for munin and mobile app: android app for munin

also plugins for PHP5-FPM, example to parse cloudwatch logs

muninmx has a nice dashboard, backend collector is written in java

and a plugin written in go, to log website response time.

but munin is certainly not the best one, it is pull model, runs by cron, which means interval is at least 1 minute. however, when you check the cron file it tells you that even the cron ran per minute, graphs are generated every 5 minutes.

it surely will have problem with large number of nodes.

howerver, for a small number of nodes (my case), it is the best one I've used.

I may check ganglia later, which scales well.

SpringDeveloper just uploads a bunch of interesting videos

I'm watching Managing secrets at scale, on using Vault by HashiCorp, pretty good one.

Tuesday, January 17, 2017

after I read the discussion on Mastering Bash and Terminal, I finally know how to jump to character using emacs binding:

(under vi-mode, f to search forward and F for backward)

nothing intersting recently, just reading Ansible for DevOps.

(notes are taken here)

I may use ansible to install puppet, puppet still the way I prefer to manage server states.

Thursday, January 19, 2017

Caching at Reddit and discussions on reddit

Mcrouter from facebook sounds useful.

Reddit's Infra/Ops team also did an AMA , quite an interesting thread.

they're also using ansible to orchestrate servers and managed by puppet, but they are switching orchestrate part to Terraform I guess.

some notes from the AMA thread:

new project has some fairly complex forms, considering using vue.js and vuex, hope they work nice with semantic-ui

also lots of suggestions from vuejs/awesome-vue

like many other people, I like vue.js simply because it doesn't require npm and webpack, what a nightmare dealing with those dev tools.

I'm happy with munin for historial graphs, and almost no maintainance effort.

however, real-time alerting system is needed as well.

I tried monit, it's nice and fits my requirement but its central management server m/monit is not free.

I ended up with icinga2, but without icingaweb2 (which requires mysql or postgresql), I hope it supports sqlite as data backend soon.

icinga2 has an api, it's not difficult to write a simple web app for a summary page. but the purpose of icinga2 is monitoring and alerting only.

cloudwatch logs can collected by munin as well: splebel/munin-aws-cloudwatch: AWS Cloudwatch plugin for Munin

munin's plugins are so simple, can easily changed to fit your needs.

some random notes:

this article about golang is nice: My Go Resolutions for 2017

and the never ending fight: Rust vs. Go

another worse is better case: RethinkDB: why we failed

rethinkdb is not dead, it becomes an open-source project. I hope it does well and I definitely will use it when I need to build a realtime app.

more bigdata tool: Apache Kudu - Fast Analytics on Fast Data

even though there're many tools for bigdata, but two things never change: HDFS and SQL

because of Reagent deep dive part 1, I decided to try reagent one more time:

$ lein new reagent-frontend myproject

comparing to lein new figwheel myproject -- --reagent, you don't have to setup cljsbuild.

and it's as compact as figwheel template: 

reagent-frontend/
├── LICENSE
├── README.md
├── env
│   ├── dev
│   │   └── cljs
│   │       └── reagent_frontend
│   │           └── dev.cljs
│   └── prod
│       └── cljs
│           └── reagent_frontend
│               └── prod.cljs
├── project.clj
├── public
│   ├── css
│   │   └── site.css
│   └── index.html
└── src
    └── reagent_frontend
        └── core.cljs

figwheel-reagent/
├── README.md
├── dev
│   └── user.clj
├── project.clj
├── resources
│   └── public
│       ├── css
│       │   └── style.css
 │       └── index.html
└── src
    └── figwheel_reagent
        └── core.cljs

however, reagent-frontend doesn't support figwheel's http server ...

lein figwheel did start a http server: http://127.0.0.1:3449, with error message Figwheel Server: Resource not found

according to reagent-frontend's project page, you'll need to open public/index.html manually.

I found the solution is not difficult, so I created a pull request to fix it:

Friday, January 20, 2017

my pull request to reagent-project/reagent-frontend-template has been accpeted and merged to master

you can download original audio file of youtube video by:

$ youtube-dl --audio-format best -x url

the format will be .m4a, .ogg or .opus most of the time. I didn't know Opus Codec before, but the quality sounds good while file size stays small.

not much players supports .opus for android, I use aimp player

on os x is simple, vox supports it.

to convert files can download opus-tools, I'll convert .opus files to .wav and play by my pocket music payer.

covert to other files to .opus is also simple:

$ ffmpeg -i in.flac out.wav
$ opusenc --bitrate 128 out.wav out.opus

Saturday, January 21, 2017

after read reddit's infrastructure, twitter also posted theirs: The Infrastructure Behind Twitter: Scale

funny that both reddit and twitter are use puppet.

I noticed there're lots of twitter csp links are blocks by disconnect add-on when reading twitter's article.

I googled and found two articles from 2013:

also funny that I read a news about csp few hour before: GitHub’s post-CSP journey

you'll need an NAT if you didn't give an internet gateway to your vpc instances, that's how aws docs told you.

but I didn't realized that a proxy can also act as a NAT.

To NAT or to Proxy, That is the Question is an good article on this.

we had a squid proxy in our stack before (that's why I never think about use a proxy as NAT), but seems nginx can do the job well too: IP Transparency and Direct Server Return with NGINX Plus (even though nginx plus is not nginx)

Kotlin 1.1 Beta Is Here!, finally kotlin comes with concurrency function: Coroutines.

don't have much time on kotlin, but will try Coroutines

this is how easy to do channels in Common Lisp, lisp is powerful.

there's another lisp language: Pixie, written by python.

syntax is similar to clojure, looks like it's another good candidate for scripting.

Discord Stores Billions of Messages in Cassandra, another read-world example that proves cassandra is reliable.

article mentioned ScyllaDB, a cassandra compatible database but written in C++, may try that.

for a long time I use evil-mode key-bindings (vi normal mode) for moving cursors, I decided to swtich back to emacs key bindings after practiced under bash for quite some time.

some new shortcuts I learned:

Sunday, January 22, 2017

someone posted The AWK Programming Language (pdf link) on hackernews

I only have one book about *awk: Sed & Awk

I should learn more awk, starting with Why you should learn just a little Awk

Thursday, January 26, 2017

reading Gawk: Effective AWK Programming, it should be the same book published by O'Reilly: Effective awk Programming: Universal Text Processing and Pattern Matching

Friday, January 27, 2017

I missed some vim keybindings after switched to emacs insert mode by default, this wiki provides some workarounds: EmacsWiki: Vi Keys

the function I was looking for is:

Sunday, January 29, 2017

reading Learning ClojureScript

in order to make M-x cider-jack-in-clojurescript work, put these to project.clj:

:profiles
  {:dev
   {:dependencies [[figwheel-sidecar "0.5.4-6"]
                   [com.cemerick/piggieback "0.2.1"]]
    :plugins      [[lein-figwheel "0.5.8"]]
    }}

and add these to init.el:

(setq cider-cljs-lein-repl
      "(do (require 'figwheel-sidecar.repl-api)
         (figwheel-sidecar.repl-api/start-figwheel!)
         (figwheel-sidecar.repl-api/cljs-repl))")

Monday, January 30, 2017

ClojureScript 1.9.456 announces two new features: Externs Inference & Comprehensive JS Modules Support

Blog Archive