jchk - /blog/2015-09

Tuesday, September 01, 2015

didn't listen to podcast for a while, here are the to-listen list:

Apache Mesos and Kubernetes are the next things I'm interested in.

the easiest way to test out kubernetes is using Google Container Engine.

Then I found out newer version of Docker (1.7) already supports multiple hosts, check Experimental: Compose, Swarm and Multi-Host Networking. which is probably good enough.

Executable Images - How to Dockerize Your Development Machine is interesting. best fit for linux desktop, on windows/mac you'll still need an vm, on linux you don't.

Wednesday, September 02, 2015

paper of mesos(pdf), and google's 3 generation of cluster management system: omega, borg, kubernetes

there're tons of tools for containers:

flocker, container data volume manager
Amazon ECS Container Agent
registrator, service registry bridge for docker
flannel, virtual network
kong, management layer for APIs
empire, control layer on top of Amazon EC2 Container Service
cadvisor, monitor containers

and services:

prometheus, service monitoring system
scout, monitoring
logentries, log management
sematext, docker monitoring

Tuesday, September 08, 2015

playing the top command:

h: help
f: fields
V: forest view
H: threads
R: sort
i: toggle idle task
L: locate string
&: find again
> and <: move sort column
x: toggle highlight for sort field
y: toggle highlight for running task
z: toggle color
B: toggle bold
A: alternative display
a: move to next field group
w: move to previous field group
g: enter field group
k: kill a pid

display non-idle with one frame: top -ibn 1

ps sort by cpu: ps aux --sort c

show memory usage in detail: pmap -x [pid]

lolcat: a colorful cat

Wednesday, September 09, 2015

trying Counterclockwise, actually I like it, on a more powerful desktop I'll use it instead of emacs.

this dark theme is good enough for me at the moment.

found this useful article when reading Linux Journal (2015 May): Hacking a Safe with Bash (on Github: windowsrefund/safe), talking about use bash to effectively encrypt / decrypt data with gpg.

reading DevOps Troubleshooting: Linux Server Best Practices, learned few things, will update it on the linux page.

tried google container engine for couple days, not quite stable. container's information doesn't response to neither web console nor cli quick enough: when removing / shutting down a node (instance), or removing a cluster, it just failed and won't do it.

19 Tips For Everyday Git Use has few useful tips:

git log -p filename to include source code changes in log output.
git log -L 1,3:filename to shows history of those lines.
git stash -p to stash partially.

about acl on ubuntu, FilePermissionsACLs, a quick note:

copy acl from one dir to another: getfacl dir1 | setfacl -b -n -M - dir2

Output from getfacl is accepted as input for setfacl when using -M, -b clear ACLs, -n do not recalculate effective rights mask, - read from stdin.

check permission recursively:

namei /path/to/file

to check permission on file/dir:

sudo -u user stat /path/to/dir/or/file

to check write permission:

sudo -u user test -w /path/to/dir/or/file; echo $?

sudo -u user test -w /path/to/dir/or/file && echo "ok" || echo "not ok"

check read and execute permission use -r and -x flags.

and for tmux, to highlight the active pane, can set both fg and bg color, which will show as a bold line for the active pane:

set-option -g pane-active-border-fg red
set-option -g pane-active-border-bg red

Thursday, September 10, 2015

read a good article about docker networking on linux journal (2015 June): Concerning Containers' Connections: on Docker Networking, but still need to check docker 1.7 for what's new on multi-host networking.

LAMP and MEAN stack, heard a new one today: the ELK stack

here's an article more about the stack: How to Deploy the ELK Stack in Production

after reading this thread on reddit: When Emacs Becomes Your Default File Manager..., I got a new command dired:

#!/bin/bash
emacsclient --eval "(dired \"${1:-~}\")"

some varnish notes: (from NGINX open sources TCP load balancing | Hacker News)

// Query Times
varnishncsa -F '%t %{VCL_Log:Backend}x %Dμs %bB %s %{Varnish:hitmiss}x "%r"'

// Slow Queries
varnishncsa -F '%t %{VCL_Log:Backend}x %Dμs %bB %s %{Varnish:hitmiss}x "%r"' -m "VCL_Log:SlowQuery"

// Top URLs
varnishtop -i RxURL

// Top Referer, User-Agent, etc.
varnishtop -i RxHeader -I Referer
varnishtop -i RxHeader -I User-Agent

// Cache Misses
varnishtop -i TxURL

// awesome dashboard
varnishstat

// to clear an url (> v4.0)
varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban "req.http.host == example.com && req.url == /"

a good article on LVM: How to Extend/Reduce LVM's (Logical Volume Management) in Linux - Part II

notes from resizing with lvm:

to test I created a 3G disk from virtualbox, and resize it to 1G.

create

$ fdisk -l
$ fdisk -c -u /dev/sdb

here:
n -> p -> 1
t -> 8e
p
w

$ pvs
$ pvcreate /dev/sdb1

$ vgs
$ vgcreate -s 32M sitedata /dev/sdb1

$ vgdisplay sitedata (get PE)
$ lvs
$ lvcreate -l 95 -n sitephotos sitedata

$ mkfs.ext4 /dev/sitedata/sitephotos
 
$ mount /dev/sitedata/sitephotos /opt/sitephotos

$ df -h

resize

$ umount /opt/sitephotos
$ e2fsck -ff /dev/sitedata/sitephotos
$ resize2fs /dev/sitedata/sitephotos 1G
$ lvreduce -L -2G /dev/sitedata/sitephotos
$ mount /dev/sitedata/sitephotos /opt/sitephotos

create from free space

$ vgdisplay sitedata (get PE)
$ lvcreate -l 64 -n sitedoc sitedata 
$ mount /dev/sitedata/sitedoc /opt/sitedoc
 
$ df -h

Tuesday, September 15, 2015

the magic ip address:

curl http://169.254.169.254/latest/meta-data/

or use ec2metadata command

Friday, September 18, 2015

reading old linux journal, found this tech tip:

# find modified files during an installation:
touch /tmp/afile
find /etc -newer /tmp/afile

# can check accessed too:
find /etc -anewer /tmp/afile

# to find between custom peroid:
touch -t 200712021345.00 /tmp/file1
touch -t 200712040130.00 /tmp/file2
find /etc -newer /tmp/file1 -a ! -newer /tmp/file2

more devops works recently, these are few intersting tools:

also more on puppet, and just created a new page for it.

to understand aws, this is a good article: AWS in Plain English

Saturday, September 19, 2015

to enable mysql compress:

SET GLOBAL innodb_file_per_table=1;
SET GLOBAL innodb_file_format=Barracuda;
ALTER TABLE mytable ROW_FORMAT=COMPRESSED;

since I didn't apply innodb_file_per_table=1 before, even I entabled and compressed tables, ibdata1 size won't shrink, and my droplet is running out of space (not enough space for a db dump)

the solution is:

create a new droplet
set up db user account and firewall rules for the mysqldump
mysqldump the database to new droplet
drop tables from original database
shutdown mysql
delete ibdata1, ib_logfile0 and ib_logfile1
start mysql, it will create a new set of ibdata1, ib_logfile0 and ib_logfile1
import from the db dump
cleanup user account, firewall rules
destroy the new droplet

Wednesday, September 23, 2015

more puppet stuffs, still don't know whether to use Defined Resource Types or Class Inheritance , I just need an elegant way to overwrite parameters.

for good style of puppet codes, as this article said, puppetlabs/puppetlabs-ntp has the almost perfect style.

Backblaze B2 provides a free 10GB cloud storage and their price is even lower than Amazon Glacier, will check their api first.

Tuesday, September 29, 2015

a quick check on mysql table sizes using a sql from this page: Finding out largest tables on MySQL Server

SELECT CONCAT(table_schema, '.', table_name),
   CONCAT(ROUND(table_rows / 1000000, 2), 'M')                                    rows,
   CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G')                    DATA,
   CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G')                   idx,
   CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
   ROUND(index_length / data_length, 2)                                           idxfrac
FROM   information_schema.TABLES
ORDER  BY data_length + index_length DESC
LIMIT  10;

Blog Archive