# Plotting Tick Data with ggplot2

Here are some examples of using ggplot2 and kdb+ together to produce some simple graphs of data stored in kdb+. I am using the qserver extension for R (http://code.kx.com/wsvn/code/cookbook_code/r/) to connect to a running kdb+ instance from within R.

First, lets create a dummy data set: a set of evenly-spaced timestamps and a random walk price series:

ONE_SEC:long$1e9 tab:([]time:.z.P+ONE_SEC * (til 1000);price:sums?[1000?1.<0.5;-1;1]) Then import the data into R: >tab <- execute(h,'select from tab') Then plot a simple line graph – remember ggplot2 works natively with data frames: >library(ggplot2) >ggplot(tab, aes(x=time, y=price)) + geom_line() + ggtitle("Stock Price Evolution") This will produce a line graph similar to the one below: Next, we can do a simple bin count / histogram on the price series: ggplot(tab, aes(x=(price))) + geom_histogram() Which will produce a graph like the following: We can adjust the bin width to get a more granular graph using the binwidth parameter: > ggplot(tab, aes(x=(price))) + geom_histogram(position="identity", binwidth=1) We can also make use of some aesthetic attributes, e.g. fill color – we can shade the histogram by the number of observations in each bin: ggplot(tab, aes(x=(price), fill=..count..)) + geom_histogram(position="identity", binwidth=1) Which results in: Some other graphs: Say I have a data frame with a bunch of currency tick data (bid/offer/mid prices). The currencies are interspersed. Here is a sample: > head(ccys) sym timestamp bid ask mid 1 AUDJPY 2013-01-15 11:00:16.127 94.485 94.496 94.4905 2 AUDJPY 2013-01-15 11:00:22.592 94.486 94.496 94.4910 3 AUDJPY 2013-01-15 11:00:30.117 94.498 94.505 94.5015 4 AUDJPY 2013-01-15 11:00:30.325 94.498 94.506 94.5020 5 AUDJPY 2013-01-15 11:00:37.118 94.499 94.507 94.5030 6 AUDJPY 2013-01-15 11:00:47.348 94.526 94.536 94.5310  I want to add a column containing the log-returns calculated separately for each currency:  log.ret <- function(x) do.call("rbind", lapply(seq_along(x), function(i) cbind(x[[i]],lr=c(0, diff(log(x[[i]]$mid))))))
ccys <- log.ret(split(ccys, f=ccys$sym))  Then I can plot a simple line chart of the log returns using the lovely facets feature in ggplot to split out a separate panel per symbol: ggplot(ccys, aes(x=timestamp, y=lr)) + geom_line() + facet_grid(sym ~ .)  Which produces the following: Another nice graph – display a visual summary of the tick frequency by time. This one uses a dummy column that represents a tick arrival. Note in the following graph I have reduced the line width and set the alpha value to a tiny value (creating a large transparency effect) as otherwise the density of tick lines is too great. The overall effect is visually pleasing: ccys <- cbind(ccys, dummy=rep(1,nrow(ccys))) ggplot(ccys, aes(x=timestamp,y=dummy, ymin=0, ymax=1)) + geom_linerange(alpha=1/2,size=.01,width=.01) + facet_grid(sym ~ .) + theme(axis.text.y=element_blank()) + xlab("ticks") + ylab("time") + ggtitle("Tick Density")  Which results in: Next time (time permitting) – covariance matrices and bivariate density plots.. # Rmathlib and kdb+ part 3: Utility Functions In the first two parts of this series, I looked at the basics of the interface I created between rmathlib and kdb+. In this post, I’ll go through some of the convenience functions I wrote to emulate some basic R functionality. NOTE: I wrote these functions as a learning exercise to familiarise myself with q/kdb+ a bit more – they are very simplistic and I am sure there are much better ways to do them (I know the very nice qml project does a lot of this stuff, plus interfaces to external BLAS libraries). Some basics: load the library: q)\l rmath.q ### seq – sequence function The first convenience function is seq, which is like R’s seq() or Numpy’s arange() in that it takes a start and end point, and generates a range of numbers. q)seq[1;100] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26... The function is just a curried wrapper around seqn, which also takes a step size: q)seqn[1;10;.5] 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10  This is handy when working with some of the probability generation functions shown last time. For instance the commands below calculate the normal distribution (cdf) values for evenly spaced points from -3 to 3: q)each[{pnorm[x;0;1]}] seqn[-3;3;.5] 0.001349898 0.006209665 0.02275013 0.0668072...  ### table – Symbol Tabulation R provides a handy function to show summary count tables of factor levels. In q, this can be used as follows: q)t:([] sym:1000?ABCD) q)table[sym;t] B| 241 A| 244 C| 256 D| 259 This tabulates the column sym from the table t. The summary is ordered by increasing frequency count. NOTE: this has really no advantage over the standard q)x xasc select count i by sym from t sym| x ---| --- B | 241 A | 244 C | 256 D | 259 except some slight brevity. ### range – Min/Max The range function simply returns the boundaries of a set of values: q)x:rnorm[10000] q)range[x] -3.685814 4.211363 q)abs (-) . range[x] / absolute range 7.897177 ### summary – Min/Max/Mean/Median/IQR The summary function provides summary stats, a la R’s summary() function: q)x:norm[10000;3;2] q)summary x min | -4.755305 1q | 1.59379 median| 2.972523 mean | 2.966736 3q | 4.336589 max | 10.00284 ### quantile – Quantile Calculations A very simple quantile calc: q)x:norm[10000;3;2] q)quantile[x;.5] 2.973137 ### hist – Bin Count Very crude bin count – specify the data and the number of bins: q)hist[x;10] -4.919383| 14 -3.279589| 101 -1.639794| 601 0 | 1856 1.639794 | 3043 3.279589 | 2696 4.919383 | 1329 6.559177 | 319 8.198972 | 40 9.838766 | 1 ### diag – Identity Matrix Generation q)diag 10 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 ### Scale functions Sometimes its very useful to scale an input data set – e.g. when feeding multiple inputs into a statistical model, large differences between the relative scales of the inputs combined with finite-precision computer arithmetic can result in some inputs being dwarfed by others. The scale function just adjusts the input as follows: $$X_{s}=(X-\mu)/\sigma$$. The example below scales two inputs with different ranges: q)x:norm[10;0;1]; y:norm[10;5;3] q)x 1.920868 -1.594028 -0.02312519 1.079606 -0.5310111 0.2762119 0.1218428 0.9584264 -0.4244091 -0.7981221 q)y 10.69666 2.357529 8.93505 3.65696 5.218461 3.246216 5.971919 7.557135 1.412827 1.246241 q)range x -1.594028 1.920868 q)range y 1.246241 10.69666 q)range scale x -1.74507 1.878671 q)range scale y -1.232262 1.845551 There are other useful scaling measures, including the min/max scale: $$\frac{x-min(x)}{max(x)-min(x)}$$. This is implemented using the minmax function: q)minmax x 1 0 0.4469273 0.760658 0.302432 0.5320897 0.4881712 0.726182 0.3327606 0.226438 range minmax x 0 1f  There are other functions which are useful for scaling, e.g. the RMSD (root-mean-square deviation): $$\sqrt{\frac{\sum{x_i^2}}{N}}$$: q)x:rnorm 1000 q)rms x 1.021065 ### nchoosek – Combinations The nchoosek function calculates the number of combinations of N items chosen k at a time (i.e. $${N}\choose{k}$$: q)nchoosek[100;5] 7.528752e+07 q)each[{nchoosek[10;x]}] seq[1;10] 10 45 120 210 252 210 120 45 10 1f The source file is available here: https://github.com/rwinston/kdb-rmathlib/blob/master/rmath_aux.q. # Rmathlib and kdb+, part 2 – Probability Distribution Functions Following on from the last post on integrating some rmathlib functionality with kdb+, here is a sample walkthrough of how some of the functionality can be used, including some of the R-style wrappers I wrote to emulate some of the most commonly-used R commands in q. ## Loading the rmath library Firstly, load the rmathlib library interface: q)\l rmath.q ## Random Number Generation R provides random number generation facilities for a number of distributions. This is provided using a single underlying uniform generator (R provides many different RNG implementations, but in the case of Rmathlib it uses a Marsaglia-multicarry type generator) and then uses different techniques to generate numbers distributed according to the selected distribution. The standard technique is inversion, where a uniformly distributed number in [0,1] is mapped using the inverse of the probability distribution function to a different distribution. This is explained very nicely in the book “Non-Uniform Random Variate Generation”, which is availble in full here: http://luc.devroye.org/rnbookindex.html. In order to make random variate generation consistent and reproducible across R and kdb+, we need to be able to seed the RNG. The default RNG in rmathlib takes two integer seeds. We can set this in an R session as follows: > .Random.seed[2:3]<-as.integer(c(123,456))  and the corresponding q command is: q)sseed[123;456] Conversely, getting the current seed value can be done using: q)gseed[] 123 456i The underlying uniform generator can be accessed using runif: q)runif[100;3;4] 3.102089 3.854157 3.369014 3.164677 3.998812 3.092924 3.381564 3.991363 3.369.. produces 100 random variates uniformly distributed between [3,4]. Then for example, normal variates can be generated: q)rnorm 10 -0.2934974 -0.334377 -0.4118473 -0.3461507 -0.9520977 0.9882516 1.633248 -0.5957762 -1.199814 0.04405314 This produces identical results in R: > rnorm(10) [1] -0.2934974 -0.3343770 -0.4118473 -0.3461507 -0.9520977 0.9882516 1.6332482 -0.5957762 -1.1998144 [10] 0.0440531  Normally-distributed variables with a distribution of $$N(\mu,\sigma)$$ can also be generated: q)dev norm[10000;3;1.5] 1.519263 q)avg norm[10000;3;1.5] 2.975766 Or we can alternatively scale a standard normal $$X ~ N(0,1)$$ using $$Y = \sigma X + \mu$$: q)x:rnorm[1000] q) int$ (avg x; dev x) 0 1i q)y:(x*3)+5 q) int$(avg y; dev y) 5 3i ## Probability Distribution Functions As well as random variate generation, rmathlib also provides other functions, e.g. the normal density function: q)dnorm[0;0;1] 0.3989423 computes the normal density at 0 for a standard normal distribution. The second and third parameters are the mean and standard deviation of the distribution. The normal distribution function is also provided: q)pnorm[0;0;1] 0.5 computes the distribution value at 0 for a standard normal (with mean and standard deviation parameters). Finally, the quantile function (the inverse of the distribution function – see the graph below – the quantile value for .99 is mapped onto the distribution function value at that point: 2.32): q)qnorm[.99;0;1] 2.326348 We can do a round-trip via pnorm() and qnorm(): q)int$ qnorm[ pnorm[3;0;1]-pnorm[-3;0;1]; 0; 1] 3i

Thats it for the distribution functions for now – rmathlib provides lots of different distributions (I have just linked in the normal and uniform functions for now. There are some other functions that I have created that I will cover in a future post.

All code is on github: https://github.com/rwinston/kdb-rmathlib

[Check out part 3 of this series]

# Integrating Rmathlib and kdb+

The R engine is usable in a variety of ways – one of the lesser-known features is that it provides a standalone math library that can be linked to from an external application. This library provides some nice functionality such as:

* Probability distribution functions (density/distribution/quantile functions);
* Random number generation for a large number of probability distributions

In order to make use of this functionality from q, I built a simple Rmathlib wrapper library. The C wrapper can be found here and is simply a set of functions that wrap the appropriate calls in Rmathlib. For example, a function to generate N randomly-generated Gaussian values using the underlying rnorm() function is:

K rnn(K n, K mu, K sigma) {
int i,count = n->i;
K ret = ktn(KF, count);
for (i = 0; i < count; ++i)
kF(ret)[i] = rnorm(mu->f, sigma->f);
return ret;
}


These have to be imported and linked from a kdb+ session, which is done using special directives (the 2: verb). I decided to automate the process of generating these directives – the code shell script below parses a set of function declarations in a delimited section of a C header file and produces the appropriate load statements:

INFILE=rmath.h
DLL=\:rmath

echo "dll:$DLL" DECLARATIONS=$(awk '/\/\/ BEGIN DECL/ {f=1;next} /\/\/ END DECL/ {f=0} f {sub(/K /,"",$0);print$0}' $INFILE) for decl in$DECLARATIONS; do
FNAME=${decl%%(*} ARGS=${decl##$FNAME} IFS=, read -r -a CMDARGS <<< "$ARGS"
echo "${FNAME}:dll 2:(\$FNAME;${#CMDARGS[*]})" done echo "\\l rmath_aux.q"  This generates a set of link commands such as the following: dll::rmath rn:dll 2:(rn;2) rnn:dll 2:(rnn;3) dn:dll 2:(dn;3) pn:dll 2:(pn;3) qn:dll 2:(qn;3) sseed:dll 2:(sseed;2) gseed:dll 2:(gseed;1) nchoosek:dll 2:(nchoosek;2)  It also generates a call to load a second q script, rmath_aux.q, which contains a bunch of q wrappers and helper functions (I will write a separate post about that later). A makefile is included which generates the shared lib (once the appropriate paths to the R source files is set) and q scripts. A sample q session looks like the following:  q) \l rmath.q q) x:rnorm 1000 / generate 1000 normal variates q) dnorm[0;0;1] / normal density at 0 for a mean 0 sd 1 distribution  The project is available on github: https://github.com/rwinston/kdb-rmathlib. Note that loading rmath.q loads the rmath dll, which in turn loads the rmathlib dll, so the rmathlib dll should be available on the dynamic library load path. [Check out Part 2 of this series] # Exporting Data From R to KDB Here is the beginnings of a simple routine to convert R data frames to Q format (in this case a dictionary). It uses the S3 dispatch mechanism to handle the conversion of different data types. Extremely basic (I havent even bothered to put in proper file output – just capturing the output of cat) but very quick to knock up. The code is mainly a top-level function called to_dict: to_dict <- function(x) { cat(substitute(x),":", sep="") nms <- names(x) for (n in nms) { cat("",n,sep="") } cat("!(",sep="") r <- rep(c(";",")"),times=c(length(nms)-1,1)) for (i in 1:length(nms)) { cat(qformat(x[[nms[i]]]),r[i],sep="") } }  Where qformat is a generic S3 function: qformat <- function(x) { UseMethod("qformat") } qformat.default <- function(x) { cat("",format(x)) } qformat.logical <- function(x) { cat(ifelse(x==TRUE,"1b","0b")) } qformat.factor <- function(x) { cat("",gsub("\\s","",format(x)), sep="") }  It can be used as follows (using the famous Anscombe quartet data): > write(capture.output(to_dict(anscombe)), file="/tmp/anscombe.q")  Then within a Q session: q)\l /tmp/anscombe.q q)anscombe x1| 10 8 13 9 11 14 6 4 12 7 5 x2| 10 8 13 9 11 14 6 4 12 7 5 x3| 10 8 13 9 11 14 6 4 12 7 5 x4| 8 8 8 8 8 8 8 19 8 8 8 y1| 8.04 6.95 7.58 8.81 8.33 9.96 7.24 4.26 10.84 4.82 5.68 y2| 9.14 8.14 8.74 8.77 9.26 8.1 6.13 3.1 9.13 7.26 4.74 y3| 7.46 6.77 12.74 7.11 7.81 8.84 6.08 5.39 8.15 6.42 5.73 y4| 6.58 5.76 7.71 8.84 8.47 7.04 5.25 12.5 5.56 7.91 6.89 # Compiling The kdb/R interface on Win32 I have been playing with the kdb/R interface from kx.com, and had some problems installing with Cygwin gcc. It may be possible to get this to work with Cygwin gcc + a Win32 threads library, but in the meantime I installed MinGW, and it works perfectly. Here are the steps (basically as per the kx docs): 1. Download c.o from here: http://kx.com/q/w32/ 2. gcc -c base.c -I. -I "${R_HOME}/include/"
3. gcc -Wl,--export-all-symbols -shared -o qserver.dll c.o base.o \${r -HOME}/bin/R.dll -lws2_32

The resulting qserver.dll can be loaded via dyn.load(), and then (just using the qserver.R supplied by kx) from within R:

source("qserver.R") conn < - open_connection("server", 12345) result <- execute(conn, "select avg bid by sym from fx_quote") x <- as.data.frame(mapply(FUN=c, result)) > head(x, 10) V1 V2 1 AUD= 0.792402880224811 2 AUD=D2 0.791632149468651 3 AUD=EBS 0.790402776387278 4 AUDCHF=R 0.85955071021153 5 AUDJPY=R 75.0707755671935 6 BRL= 1.97194091379422 7 CAD= 1.15980648929715 8 CAD=D2 1.15962545479939 9 CAD=EBS 1.14104373919176 10 CADJPY=R 81.6389284332255 `