Tuesday, November 07, 2017

Rich Hickey's talk: Effective Programs - 10 Years of Clojure (video, transcript, discussions on hackernews) brings lots of discussions about types

and I think this is also related:


couple new tools for clojure:

and again, Clojure transducers from the ground up: using them in practice.


some good reads from past couple weeks:


after read Learn to use Awk with hundreds of examples and Why Learn AWK?

I'm reading The AWK Programming Language (download pdf from archive.org) again


start coding in python, because all my teammates are python people

surely emacs has great support on *python, install elpy, company-jedi packages

then I can eval the buffer by c-c c-c, just like cider, (more on elpy)

also support virtualenv, once you created a virtualenv folder in your project, use m-x pyvenv-activate to select it and your repl now works within the env.


a snippet for catch python exceptions with traceback:

import logging

logging.basicConfig(filename='/tmp/out.log', level=logging.ERROR)

try:
    run_my_stuff()
except:
    logging.exception('Got exception on main handler')

I'm learning to write good python codes as well:

Monday, November 20, 2017

a awk script to extract slow queries:

awk '/Query_time: [2-9]{2,}/{a=1;print "";print NR;print l;print;next}/Query_time/{a=0}/Id:/{l=$0;next}a>0' mysql-slowquery.log


some notes when learning postgresql:

switch to postgres user:

> sudo -iu postgres

  • > createuser –interactive
  • > dropuser jim

By default, another assumption that the Postgres authentication system makes is that there will be an database with the same name as the role being used to login, which the role has access to.

  • > createdb jim
  • > dropdb jim

include sql file # \i /path/to/file.sql

describe tables

  • # \d
  • # \dt (without seq tables)
  • # \d+ table_name (like desc table_name)

import from csv file:

# copy table_name from '/path/to/csv_file' CSV HEADER;

to use crosstab, first enable tablefunc:

# CREATE extension tablefunc;

The SELECT parameter feed to crosstab must return 3 columns:

  1. as identifier
  2. as category
  3. value

In practice the SQL query should always specify ORDER BY 1,2 to ensure that the input rows are properly ordered

use the crosstab function with a second parameter, which represents the complete list of categories.

export to csv

COPY (select id,name from tablename) TO 'filepath/aa.csv' DELIMITER ',' CSV HEADER;

reference:


I also start learning jupyter (ipython, pandas etc.)

start: jupyter notebook

  • edit mode: green border
  • command mode: blue border

shortcuts:

  • Esc: switch to command mode
  • [Shift+Enter]: run that cell (and select the cell below it; or, use [Control+Enter] if you want to stay in that cell); this works even if you are in Edit Mode
  • x: delete that cell
  • a: create a new cell above
  • b: create a new cell below
  • m: make that cell a markdown cell
  • l: add line numbers (useful when working with others)
  • y: make that cell a code cell (which is the default for a new cell)
  • [Enter]: enter Edit Mode (turning the cell green and putting the cursor inside)
  • ↑ ↓: use the up and down arrows to move through the cells
  • [Number]: type a number (i.e., 1, 2, 3,…) to add a heading with that size (then use Enter to get into Edit Mode)

for db password, use dotenv

  1. pip3 install python-dotenv
  2. %load_ext dotenv
  3. %dotenv

pandas

10 Minutes to pandas

axis:

+------------+---------+--------+
|            |  A      |  B     |
+------------+---------+---------
|      0     | 0.626386| 1.52325|----axis=1----->
+------------+---------+--------+
                |         |
                | axis=0  |
                ↓         ↓

Blog Archives

Search Blog: