Jim Cheung

Saturday, January 02, 2016

Clojure 2015 Year in Review, in 2015 I also started porting my projects to clojure.

in the HN thread found Leviathan, a clojure IDE for os x.

Sunday, January 03, 2016

to install Kodi on ubuntu

sudo add-apt-repository ppa:team-xbmc/ppa
sudo apt-get update
sudo apt-get install kodi

launch from launcher: Kodi Media Center

one simple way to share files to Kodi is via nfs

sudo apt-get install nfs-kernel-server

add to /etc/exports:

/export/share *(ro,all_squash,insecure,no_subtree_check)

one tool to monitor network traffic is nethogs

Tuesday, January 05, 2016

reading The Varnish Book, actually it's not bad, I quite like it.

also reading DevOps Automation Cookbook, it covers few topics that I'm interested in: aptly, influxDB and grafana, from rsyslog to ELK stack, and finally sensu. pretty much all related to my current daily tasks, but like many other packt books, it just scratch the surface of each tool.

some updates from tweets:

I did try to play with reagent, but the project.clj is way too complicated and doesn't make any sense to me. I think I need to take a look at boot first.

two upcoming books that I will read:

when I was debugging a syslog-ng issue, learned quite a lot network tools:

netcat is very powerful:

transfer file:

# server 2
nc -l 3721 > target

# server 1
nc -w 1 server2 3721 < source

clone the whole drive:

# server 2
nc -l 3721 | dd of=/dev/sdf

# server 1
dd if=/dev/sda | nc server2 3721

server a webpage (include response headers inside the html file):

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8

<html>
...
</html>

then:

while true; do nc -l 80 -q 1 < index.html; done

as http client:

nc example.com 80

chat:

# server 2
nc -l 3721

# server 1
nc server2 3721

to transfer sensitive data, can encrypt it first

# server 2
nc -l 3721 | gpg -d

# server 1
gpg -ca < source | nc server2 3721

Wednesday, January 06, 2016

podcast episodes to listen:

new podcast: Three Devs and a Maybe: Discussion on Web Development

reading Mastering Linux Network Administration, quite good.

Friday, January 08, 2016

podcasts to listen:

mytop is a nice program to monitor mysql processes: How To Use Mytop to Monitor MySQL Performance

finished Mastering Linux Network Administration, upload the notes here later.

now reading Amazon Web Services in Action, not as deep as I expected, almost finished ... maybe this one is better.

read a lot about puppet's environment setup, will upload the notes later as well.

Monday, January 11, 2016

How to Name Clojure Functions is quite good.

reading Practical Linux Topics.

Friday, January 15, 2016

there is a openssh client bug, add this to .ssh/config until the client is patched:

Host *
  UseRoaming no

not much interesting thing I could find in Practical Linux Topics, actually I read too many linux books these days. I should go back to read my clojure books.

also finished The Varnish Book, but seems The Varnish Users Guide is more useful.

I don't think I'll read Joy of Clojure and Clojure in Action page by page, in the mean time probably will start reading one of these:

looking for a simple queue worker, don't want any dependencies since, find one using shell script from github, perfect: sqs-worker.sh

#!/bin/bash

QUEUE=

echo "starting amazon sqs worker for queue $QUEUE"

while true
do
    MESSAGE=`aws sqs receive-message --wait-time-seconds=1 --visibility-timeout=600 --queue-url=$QUEUE`
    if [ $? -ne 0 ]; then
        echo "aws sqs operation failed, trying again in 5s"
        sleep 5
        continue
    fi

    NUM_MESSAGES=`echo $MESSAGE | jq '.Messages | length'`
    if [ "$NUM_MESSAGES" -eq 0 ]; then
        sleep 10
        continue
    fi

    HANDLE=`echo $MESSAGE  | jq -r '.Messages[0].ReceiptHandle'`
    TOPIC=`echo $MESSAGE | jq -r '.Messages[0].Body | fromjson | .TopicArn'`
    echo "message received on topic $TOPIC"

    # run a process
        
    if [ $? -ne 0 ]; then
        echo "error processing message"
    else
        aws sqs delete-message --queue-url=$QUEUE --receipt-handle=$HANDLE
    fi
done

learned two tricks about history: (history Man Page)

# clear current session's history
$ history -c

# append current sessions's history to file now
history -a

Monday, January 18, 2016

to generate token from desktop, can use brew to install OATH Toolkit (brew install oath-toolkit, ubuntu just apt-get install oathtool), then run the oathtool command:

cat secret_key | xargs oathtool --totp -b

Tuesday, January 19, 2016

another combination that I like: Oden: lisp + go, playing with it.

to avoid gpg prompt for confirming when encrypting a file, can use --trust-model=always:

gpg -e -r [email] --trust-model=always [file]

Thursday, January 21, 2016

for password management:

also read: Adding Yubikey 2-factor authentication to SSH and sudo in Debian

Friday, January 22, 2016

probably the most comprehensive guide for Drupal Performance Optimization, some useful links from the article:

however the varnish links are old and not for varnish 4.

the recent 2 ruby rogues podcasts are quite good:

after read this article: Datomic: The most innovative DB you've never heard of, I really want to try Datomic now.

Saturday, January 23, 2016

start playing with Datomic, will put notes to the datomic page.

also updating the nginx and varnish notes.

Monday, January 25, 2016

saw a folder organizer on github, it's a good idea but I'll prefer do it myself, by clojure.

for php-fpm, process manager value choice between dynamic and ondemand, actually I don't know yet until I do some tests.

Tuesday, January 26, 2016

working on fine tuning the php-fpm, I didn't see much difference between dynamic and ondemand, but since ondemand has fewer settings to tune, I picked ondemand.

for ondemand only two settings to tune: pm.max_children and pm.max_requests.

there're articles about how to get the pm.max_children value, first get the average memory usage of a php-fpm process:

ps -ylC php-fpm --sort:rss

depends on how many memories reserved for other services, available memory / average memory per process to get the pm.max_children value.

however I just use the number as reference and test with siege to find the number that I want.

I also didn't change the ulimit -n number since there is an option rlimit_files to handle it, leave it as default until it becomes the bottle neck.

for nginx, worker_processes always set to as same as cpu cores, not number of cores * 2, tune the worker_connections (usually >= 1024), and:

events {
  use epoll;
  multi_accept on;
}

for the reason can check my nginx notes

playing some browsers. Midori is the best, very small but can render complex pages well. for terminal, ELinks is good.

Wednesday, January 27, 2016

for nginx, if the web server is behind aws elastic load balancer, use this line to get read client ip for access control:

set_real_ip_from 0.0.0.0/0;
real_ip_header X-Forwarded-For;

Friday, January 29, 2016

a simple way to enhance clojure repl is use rlwrap, I tried jline, but it's too difficult (jline2 doesn't even work).

> sudo apt-get install -y rlwrap
> rlwrap java -jar clojure-1.8.0.jar

I prefer this way because it is much much faster than a lein repl and boot repl

for dependencies, not easy to hot-load the libraries into the repl. I will setup a lein project.clj and use lein-libdir to copy all jars into the lib folder, and add it to the classpath:

first add lein-libdir to ~/.lein/profiles.clj:

{:user {:plugins [[lein-libdir "0.1.1"]]}}

add dependencies to project.clj, then run 

> lein libdir

this task will copy all jars into ./lib folder.

then can start repl with:

> rlwrap java -cp clojure.1.8.0.jar:lib/* clojure.main

Clojure 1.8.0
user=> (require '[clj-http.client :as curl])
user=> (curl/head "http://www.google.com")

now can use clojure as shell script too, first create a simple script, name it as /usr/local/bin/run-clj:

#!/bin/sh

java -cp /opt/clojure/clojure.1.8.0.jar:/opt/myclj/lib/* clojure.main "$@"

create this as example.clj:

#!/usr/local/bin/run-clj

(require '[clj-http.client :as curl])
(clojure.pprint/pprint (curl/head "http://www.google.com"))

now simply run ./example.clj.

Sunday, January 31, 2016

starting a new project, a text base UI bot program. like most other projects, research and pick some libraries first. I miss the days when started with nothing and just write everything myself, crappy but lots of fun.

couple notes:

Blog Archive