Jim Cheung

Tuesday, February 09, 2016

to dump a sample set of data from mysql

mysqldump --user=user_name --password --opt --where="1 limit 100" table_name > sample.sql

Friday, February 12, 2016

emacs' window splitting behaviour has been bothered me for a long time, like mentioned in this article: Better window splitting in Emacs

I change the final expression so I can start selecting files right after switched:

(defun usrj/vsplit ()
  (interactive)
  (split-window-vertically)
  (other-window 1 nil)
  (ido-find-file))

(defun usrj/split ()
  (interactive)
  (split-window-horizontally)
  (other-window 1 nil)
  (ido-find-file))

gave Cursive a try, it's quite nice but I still prefer emacs at this moment.

Wednesday, February 17, 2016

reading The Go Programming Language and Clojure Applied.

the go book is really good, the clojure book is practical. I think I'll stick with clojure.

setting up a ELK stack, start with Filebeat, and learn how to parse logs with Logstash, and the elastic stack is very complete than few years ago.

Thursday, February 18, 2016

os x can not use private key generated on windows by putty. need a conversion first:

> brew install putty
> puttygen privatekey.ppk -O private-openssh -o privatekey.pem

ps aux contains long commands that won't fit in a screen, pipe it to less to see the full command. also for the state code (those Ss, S+ etc.):

PROCESS STATE CODES
       Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process:
       D    uninterruptible sleep (usually IO)
       R    running or runnable (on run queue)
       S    interruptible sleep (waiting for an event to complete)
       T    stopped, either by a job control signal or because it is being traced.
       W    paging (not valid since the 2.6.xx kernel)
       X    dead (should never be seen)
       Z    defunct ("zombie") process, terminated but not reaped by its parent.

       For BSD formats and when the stat keyword is used, additional characters may be displayed:
       <    high-priority (not nice to other users)
       N    low-priority (nice to other users)
       L    has pages locked into memory (for real-time and custom IO)
       s    is a session leader
       l    is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
       +    is in the foreground process group.

today Go 1.6 is released.

when I upgraded and restarted my server, docker containers were not able to start.

found the reason is the ip address of docker changed from 172.17.42.1 to 172.17.0.1, same as mentioned here: 1.9: Default docker0 IP is not 172.17.42.1 for new installs

the solution is update /etc/default/docker by adding this line:

DOCKER_OPTS="--bip 172.17.42.1/16"

I hate it, and I don't like docker messes up with df -h

Remotely Exploitable GlibC DNS Bug Discovered

one note to the above GlibC bug, can use lsof -n | grep DEL to check any services still using the old glibc (if not gonna reboot)

some quick notes from Clojure Applied:

Tuesday, February 23, 2016

read Learning ELK Stack, good introduction.

even if you're not going to setupt the full elk stack, just logstash is quite useful for log forwarding and organizing in a centralized palce, then you can use classic unix tools to do some simple analyzing.

The Lumberjack Protocol seems the recommended format for log forwarding, the official filebeat look promising. and there're tons of Output plugins.

for a quick daemon process, can use nohup

$ nohup myprogram </dev/null >myprogram.log 2>&1 &

to daemonize the program, stdin, stdout and stderr must be disconnected, that's what </dev/null >myprogram.log 2>&1 does.

can combine with nice:

$ nohup nice -n -5 myprogram </dev/null >myprogram.log 2>&1 &

ubuntu's start-stop-daemon also convenient way to do it:

# start
$ start-stop-daemon -S -b -x /path/to/myprogram

# stop
$ start-stop-daemon -K -n myprogram

another way is copy and modify /etc/init.d/skeleton, but it's not lazy enough for me.

and for docker there is docker run --restart=always.

Unix Programming FAQ describes how to make your program act like a daemon.

I think I'll read some chapters in Advanced Programming in the UNIX Environment and Advanced UNIX Programming, quite interested in it.

Wednesday, February 24, 2016

I'm playing slack api by writing a robot. My colleague wrote one with go, but I really have problem reading go codes. I'm gonna use clojure.

a robot basically is a program that monitor the Real Time Messaging API and response to certain events. google slack rtm client you can have many examples.

I'm using a very simple one: casidiablo/slack-rtm, simple but it works. just one note on closing the connection, which couldn't find on readme:

(send-event dispatcher :close)

one thing I don't like go is they don't have default config format which supported by core library. in clojure is very simple:

;; put data in a map
(def config {:token "abc" :channel "#general"})

;; output edn, save it to the config file
(prn config)
;; {:token "abc", :channel "#general"}
;; nil

;; (clojure.edn/read-string (slurp "config.file"))
;; {:token "abc", :channel "#general"}

more about edn-format

Thursday, February 25, 2016

reading Advanced UNIX Programming, it's so good. actually this is the book that every linux user should read.

That Linux flaw may be fixed, but what about your containers? reminds me I should upgrade all my containers too. lucky one of the practice I followed is use a custom base image, after upgraded the base image, re-built all the rest images.

continue building the slack bot, using clojure means I can eval s-expression very easily, that bring so much power to the bot.

one thing I like to put on my website is random quotes/jokes, will be nice to add them to my bot.

Friday, February 26, 2016

install ELK stack (ubuntu 14.04)

$ curl https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

put following to /etc/apt/sources.list.d/elk.list:

deb http://packages.elastic.co/elasticsearch/2.x/debian stable main
deb http://packages.elastic.co/logstash/2.2/debian stable main
deb http://packages.elastic.co/kibana/4.4/debian stable main

$ sudo apt-get update && sudo apt-get install elasticsearch logstash kibana

and put them on boot:

$ sudo update-rc.d elasticsearch defaults 95 10
$ sudo update-rc.d logstash defaults 95 10
$ sudo update-rc.d kibana defaults 95 10

config the logstash to receive logs

$ install logstash-input-beats

add following to /etc/logstash/conf.d/config.json:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

install a web admin panel for elasticsearch

$ /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf

kopf is available at http://localhost:9200/_plugin/kopf

and for kibana, edit /opt/kibana/conf/kibana.yml for host, port and elasticsearch address.

config the log forwarder,

$ curl https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/beats/apt stable main" |  sudo tee -a /etc/apt/sources.list.d/beats.list

$ sudo apt-get update && sudo apt-get install filebeat
$ sudo update-rc.d filebeat defaults 95 10

change /etc/filebeat/filebeat.yml, default is forwarding to elasticsearch, comment it and uncomment the logstash part.

to debug ./filebeat -c filebeat.yml -e -d "*"

Sunday, February 28, 2016

to make the eval of the slack bot more secure, I changed to clojail.

create a ~/.java.policy:

grant {
  permission java.security.AllPermission;
};

the usage is very simple:

(def sb (sandbox secure-tester-without-def :timeout 5000))

(sb '(+ 1 2))
;; 3

Monday, February 29, 2016

Happy February 29th!

slack webui sucks so bad on my firefox, so I started searching for a cli client. TidalLabs/Slacker is interesting, PHP + ncurses, good reference.

for ncurses and java, mabe02/lanterna looks good.

updated my marksix function: 

(defn marksix [] #(loop [s #{}] (if (< (count s) 6) (->> 48 rand-int inc (conj s) recur) (sort s))))

for the slack bot I'll implement/copy some of the commands from Raynes/lazybot. also I want to see how their plugin system works.

Blog Archive