tag:blogger.com,1999:blog-49464908068485698402024-03-13T08:48:27.795-07:00R snippetsA collection of short R code samples I find reusable and interesting.Unknownnoreply@blogger.comBlogger76125tag:blogger.com,1999:blog-4946490806848569840.post-49408828057414305332017-06-27T13:54:00.001-07:002017-06-27T22:11:15.010-07:00Julia vs R and Python: what does Stack Overflow Developer Survey 2017 tell us?TLDR: Most Julia programmers also use Python. However, among all languages R is the one whose users are most likely to also develop in Julia.<br />
<br />
Recently Stack Overflow has made public the <a href="https://stackoverflow.blog/2017/06/15/download-stack-overflows-2017-developer-survey-data/" target="_blank">results of Developer Survey 2017</a>. It is definitely an interesting data set. In this post I analyzed the answers to the question "<i>Which of the following languages have you done extensive development work in over the past year, and which do you want to work in over the next year?</i>" from the perspective of Julia language against other programming languages.<br />
<br />
Actually we get two variables of interest: 1) what was used and 2) what is planned to be used.<br />
<br />
<h4>
Basic statistics</h4>
Let us start off with raw Julia statistics: 0.18% developers reported having used Julia so it is not much. But 0.67% say that they want to use it next year, which is quite a large expected growth.<br />
<br />
As usually developers use several languages, especially for data science, there are two natural questions: 1) what other languages Julia programmers also use and 2) users of what languages also use Julia.<br />
<br />
<h4>
Languages mostly used by Julia programmers</h4>
Last year:<br />
<br />
<ol>
<li><span style="font-family: "courier new" , "courier" , monospace;">Python 73.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">JavaScript 40.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">SQL 39.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">C++ 36.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">C 34.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Java 32.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">R 29.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Matlab 21.0%</span></li>
</ol>
<br />
Plans for next year<br />
<br />
<ol>
<li><span style="font-family: "courier new" , "courier" , monospace;">Python 64.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">R 38.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">JavaScript 29.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">SQL 26.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Go 26.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">C++ 26.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Scala 24.0%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Haskell 21.0%</span></li>
</ol>
<br />
We can see that Python is the most popular language among Julia developers. However, we can see that R can be expected to grow significantly in popularity in the near future.<br />
<br />
<h4>
Users of what languages also use Julia</h4>
Last year<br />
<br />
<ol>
<li><span style="font-family: "courier new" , "courier" , monospace;">R 2.26%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Python 0.79%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">C++ 0.42%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">SQL 0.35%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">JavaScript 0.27%</span></li>
</ol>
<br />
<div>
Plans for next year</div>
<br />
<ol>
<li><span style="font-family: "courier new" , "courier" , monospace;">R 8.73%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">Python 2.70%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">C++ 1.72%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">SQL 1.64%</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">JavaScript 1.09%</span></li>
</ol>
<br />
Interestingly here the situation is reversed. R developer is much more likely to use Julia than Python programmer. The situation is similar for last year and for plans for the coming year.<br />
<br />
<a href="https://gist.github.com/bkamins/f6564d4c45ff4c9d2ad79b16872d50fc" target="_blank">Here</a> you can get the code I used to produce the above results.Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-4946490806848569840.post-30063015001003711112017-06-08T06:27:00.003-07:002017-06-08T06:54:59.542-07:00Test-driving Microsoft Cognitive Toolkit in R using reticulateRecently new tools for data science pop up constantly, so it is hard to keep up with the changes and choose those that promise to be useful in the long run.<br />
Recently two new solutions were announced that look very promising: <a href="https://github.com/rstudio/reticulate" target="_blank">reticulate</a> package for R and <a href="https://github.com/Microsoft/CNTK" target="_blank">Microsoft Cognitive Toolkit 2.0</a> (CNTK). Together with <a href="https://www.linkedin.com/in/jakuczunwit/" target="_blank">Wit Jakuczun</a> we have decided to test drive them on <a href="https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-data-science-virtual-machine-overview" target="_blank">Azure Data Science Virtual Machine</a>.<br />
CNTK is a very promising deep learning toolkit, backed by Microsoft. Unfortunately it provides bindings to Python and not to R. Therefore it is an ideal situation to test reticulate package, which provides R interface to Python. We wanted to test CNTK on GPUs, so we have decided to use Azure Data Science Virtual Machine using NV6 Instance with Tesla M60 GPU.<br />
For our test-drive we have used classic MNIST dataset. You can find all the source codes and detailed results of the test on <a href="https://github.com/WLOGSolutions/microsoft_cntk2.0_from_r" target="_blank">GitHub</a>. The short conclusion is that all components worked and played together excellently:<br />
<br />
<ul>
<li>on CNTK we have build MLP network model, which has reported 2.3% classification error, without any special tuning, which is a pretty decent result;</li>
<li>calling Python library from R using reticulate was simple and stable;</li>
<li>Calculations using GPU give a significant speedup and configuration of Azure DSVM was really simple.</li>
</ul>
<div>
In summary, we have found the combo MSFT CNTK + R/reticulate + Azure DSVM to be simple to set up and powerful. It if definitely worth checking in your next data science project.</div>
<br />
<br />Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4946490806848569840.post-54363947833645175332016-12-05T16:01:00.000-08:002016-12-05T16:01:03.235-08:00Optimization matchup: R's glpkAPI vs Julia's JuMPtl;dr: although I use R every day and love it, doing mathematical programming using Julia is much simpler and more flexible than anything I know that is currently available in R.<br />
<br />
Recently I have learned that Iain Dunning and Joey Huchette and Miles Lubin have received 2016 INFORMS Computing Society prize for the development of JuMP, a Julia-based domain-specific modeling language. Congratulations!<br />
<br />
Together with Wit Jakuczun we have decided to test drive Julia's JuMP against R's package glpkAPI.<br />
<br />
The problem we decided to solve is a standard MIP model for finding clusters in data using k-medoids method (we have used <a href="https://cseweb.ucsd.edu/~dasgupta/291-unsup/lec3.pdf" target="_blank">this</a> specification of the model without relaxation). The work breakdown was that Wit writes a solution in R and I developed Julia code.<br />
<br />
<h3>
Data preparation</h3>
The data set used for the analysis can be generated using this Julia code:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">function gendata(N, K, scaling)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x = randn(N, 2)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:N</span><br />
<span style="font-family: Courier New, Courier, monospace;"> α = 2π * i / K</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x[i, :] += scaling * [cos(α); sin(α)] / (2 * sin(π/K))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return x</span><br />
<span style="font-family: Courier New, Courier, monospace;">end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">srand(1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">d = gendata(50, 4, 5)</span><br />
<span style="font-family: Courier New, Courier, monospace;">writecsv("test.txt", d)</span><br />
<br />
or similar R code respectively:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">gen_data <- function(N, K, scaling) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> alpha <- 2 * pi * (1:N) / K</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sin_pi_K <- 2 * sin(pi / K)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> X <- as.data.frame(matrix(data=rnorm(n=2*N), nrow=N, ncol=2) +</span><br />
<span style="font-family: Courier New, Courier, monospace;"> scaling*matrix(data = c(cos(alpha), sin(alpha)),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> nrow = N, ncol = 2) / sin_pi_K)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return(data.frame(x = X$V1, y = X$V2))</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">set.seed(1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">write.table(x = gen_data(200, 4, 5), file = "test.txt",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> col.names = TRUE, row.names = FALSE,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sep = ",", dec = ".")</span><br />
<br />
(Julia codes below assume that data set was generated using Julia and R assumes data was generated in R: the difference is that file generated in R has a header)<br />
<br />
Comparing the above codes (although they are not the main objective of the exercise) highlights two things about Julia: 1) you can use unicode literals in your code and 2) in Julia using explicit loop is OK (fast). Those two points are in this case of course only an issue of taste but I actually find that both of them make the code a bit more readable.<br />
<br />
The end result is 200 points placed in 2D plane in four clusters.<br />
<br />
<h3>
Now we move to the actual challenge: <i>write a code that executes k-medoids algorithm for number of clusters from 2 to 6 and compare the results</i>.</h3>
Here is the solution in Julia:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">using JuMP</span><br />
<span style="font-family: Courier New, Courier, monospace;">using GLPKMathProgInterface</span><br />
<span style="font-family: Courier New, Courier, monospace;">using Distances</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">function find_clusters(K, ds)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> N = size(ds, 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> rho = pairwise(Euclidean(), ds')</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> m = Model(solver=GLPKSolverMIP())</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @variable(m, y[1:N], Bin)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @variable(m, x[1:N,1:N], Bin)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> @objective(m, Min, sum{x[i,j]*rho[i,j], i=1:N, j=1:N})</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> @constraint(m, sum(y) == K)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:N</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @constraint(m, sum(x[i,:]) == 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:N, j in 1:N</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @constraint(m, x[i,j] <= y[j])</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> solve(m)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return getvalue(x), getvalue(y), getobjectivevalue(m)</span><br />
<span style="font-family: Courier New, Courier, monospace;">end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">function exec()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ds = readcsv("test.txt")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> N = size(ds, 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for K in 6:-1:2</span><br />
<span style="font-family: Courier New, Courier, monospace;"> println("--- K=$K ---")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> @time x, y, obj = find_clusters(K, ds)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> println("Objective: $obj")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> println("Centers:")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:N</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y[i] > 0.0 && println("\t$i:\t", ds[i,:])</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;">end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">exec()</span><br />
<br />
and here is the R code:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">library(glpkAPI)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">find_clusters <- function(K, ds) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> N <- nrow(ds)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> rho <- cbind(expand.grid(i = 1:N, j = 1:N),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> dist = as.numeric(as.matrix(dist(ds, upper=TRUE, diag=TRUE))))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> write.table(x = rho, </span><span style="font-family: "Courier New", Courier, monospace;">file = "dist.csv",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sep = ",", </span><span style="font-family: "Courier New", Courier, monospace;">dec = ".",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> col.names = TRUE, </span><span style="font-family: "Courier New", Courier, monospace;">row.names = FALSE)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> #dat file</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat("data;\n", file = "kmedoids.dat", append = FALSE)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat(sprintf("param pN := %s;\n", N),file="kmedoids.dat", append=T)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat(sprintf("param pK := %s;\n", K),file="kmedoids.dat", append=T)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat("end;", file = "kmedoids.dat", append = TRUE)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> wk <- mplAllocWkspGLPK()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> termOutGLPK(GLP_OFF)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> model <- mplReadModelGLPK(wk, "kmedoids.mod", TRUE)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> data <- mplReadDataGLPK(wk, "kmedoids.dat")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> mplGenerateGLPK(wk)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> lp <- initProbGLPK()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> mplBuildProbGLPK(wk, lp)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> scaleProbGLPK(lp, GLP_SF_AUTO)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> setDefaultMIPParmGLPK()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> setSimplexParmGLPK(parm = c(MSG_LEV), val = c(GLP_MSG_OFF))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> setMIPParmGLPK(parm = c(MSG_LEV), val = c(GLP_MSG_OFF))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> solveSimplexGLPK(lp)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> solveMIPGLPK(lp) </span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> obj <- mipObjValGLPK(lp)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> all_values <- mipColsValGLPK(lp)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> all_names <- rep(NA_character_, length(all_values))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (i in 1:length(all_values)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> all_names[i] <- getColNameGLPK(lp, i)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sol <- data.frame(var = all_names, val = all_values)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> return(list(</span><br />
<span style="font-family: Courier New, Courier, monospace;"> obj = obj,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y = sol[grep(pattern = "vSegmentCenter", x = all_names), ],</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x = sol[grep(pattern = "vPointToSegment", x = all_names), ]))</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">ds <- read.table(file = "test.txt", </span><span style="font-family: "Courier New", Courier, monospace;">sep = ",",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> dec = ".", </span><span style="font-family: "Courier New", Courier, monospace;">header = TRUE)</span><br />
<span style="font-family: Courier New, Courier, monospace;">N <- nrow(ds)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for (K in seq(6,2)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print(sprintf("--- K = %s ---", K))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> start_time <- Sys.time()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sol <- find_clusters(K, ds)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> calc_duration <- Sys.time() - start_time</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print(sprintf(" %s sec", calc_duration))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print(sprintf("Objective: %s", sol$obj))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print("Centers:")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print(ds[sol$y$val > 0, ])</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
which requires auxiliary kmedoids.mod file:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">param pN;</span><br />
<span style="font-family: Courier New, Courier, monospace;">param pK;</span><br />
<span style="font-family: Courier New, Courier, monospace;">set pPoints := 1..pN;</span><br />
<span style="font-family: Courier New, Courier, monospace;">set pSegments := 1..pK;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">param pDist{i in pPoints, j in pPoints} >= 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">table T_dist IN "CSV" "dist.csv":</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>[i,j], pDist ~ dist;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">var vPointToSegment{i in pPoints, j in pPoints}, binary;</span><br />
<span style="font-family: Courier New, Courier, monospace;">var vSegmentCenter{i in pPoints}, binary;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">minimize total_cost: sum{i in pPoints, j in pPoints} (vPointToSegment[i,j] * pDist[i,j]);</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">subject to</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">cPointToOnlyOneSegment{i in pPoints}:</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>sum{j in pPoints} vPointToSegment[i,j] = 1;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">cSegmentsCnt:</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>sum{i in pPoints} vSegmentCenter[i] = pK;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">cPoinOnlyToActiveSegment_1{i in pPoints, j in pPoints}:</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>vPointToSegment[i, j] <= vSegmentCenter[j];</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">cPoinOnlyToActiveSegment_2{j in pPoints}:</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>sum{i in pPoints} vPointToSegment[i, j] <= card(pPoints)*vSegmentCenter[j];</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">end;</span><br />
<br />
I do not want to go through the code in detail - please judge it yourself. However, the two things are immediately apparent in my opinion:<br />
<br />
<ol>
<li>the ability to be able to specify the model within Julia beats hands down glpkAPI, where you have to use external file for specification of the model and external files to communicate data to the model;</li>
<li>in Julia if we want to switch the solver from GLPK to any other (eg. CBC) the only line we would have to change is <span style="font-family: "Courier New", Courier, monospace;">m = Model(solver=GLPKSolverMIP())</span>. In R you have bindings to other solvers but they are much more low-level and most importantly you would have to rewrite most of the code.</li>
</ol>
Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-4946490806848569840.post-41041745750627963122016-10-19T00:03:00.000-07:002016-10-19T00:03:04.420-07:00Deep learning in the cloud with MXNetLast Friday together with <a href="https://www.linkedin.com/in/przemys%C5%82aw-szufel-720a4411b" target="_blank">Przemysław Szufel</a> and <a href="https://www.linkedin.com/in/jakuczunwit" target="_blank">Wit Jakuczun</a> we were giving a live demo on introduction to deep learning at <a href="http://www.digitalchampions.pl/" target="_blank">Digital Champions</a> conference.<br />
<br />
The objective of the workshop was to show how to build a simple predictive model using MXNet library in a few minutes. In the example, we train a model on GPUs using a standard <span style="background-color: white;">MNIST database of handwritten digits.</span><br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white;">Our post-presentation thoughts are that many people did not have experience with using GPUs in the cloud. Our point is that it is actually simpler and cheaper than having the infrastructure on-premises. Running of the whole tutorial cost us under 1$. Using </span>bitfusion.io <a href="https://aws.amazon.com/marketplace/seller-profile?id=3b372560-86bf-4e3d-9ec0-016892a64bed" target="_blank">Scientific Computing AMI</a> we were able to painlessly get the whole computing environment running on AWS in a few minutes.<br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white;">On <a href="https://github.com/WLOGSolutions/digital_champions_deeplearning_r_mxnet" target="_blank">GitHub</a> you can find the instructions how to set up computations in the cloud and the R source code allowing to build the model using MXNnet.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-62961512981806350972016-09-30T00:21:00.001-07:002016-09-30T00:21:16.991-07:00R+H2O for marketing campaign modelingMy last post about <a href="http://rsnippets.blogspot.com/2016/09/telco-churn-prediction-with-rh2o.html">telco churn prediction with R+H2O</a> attracted unexpectedly high response. It seems that R+H2O combo has currently a very good momentum :). Therefore <a href="https://www.linkedin.com/in/jakuczunwit">Wit Jakuczun</a> decided to publish a case study that he uses in his <i>R boot camps</i> that is based on the same technology stack.<br />
<br />
You can find the shortened version of the case study material along with the source codes and data sets on <a href="https://github.com/WLOGSolutions/retail-segmentation-based-marketing-campaing-in-r-and-h2o/">GitHub</a>.<br />
<br />
I think it is nice not only because it is another example how to use H2O in R, but it is also a basic introduction to how to combine segmentation and prediction modeling for marketing campaign targeting.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4946490806848569840.post-17834888757296589472016-09-12T03:04:00.000-07:002016-09-12T10:11:01.281-07:00Telco churn prediction with R+H2ORecently together with my friend <a href="https://www.linkedin.com/in/jakuczunwit">Wit Jakuczun</a> we have discussed about a <a href="http://blog.revolutionanalytics.com/2016/08/telco-customer-churn-with-r-in-sql-server-2016.html">blog post</a> on Revolution showing application of SQL Server R services to build and run telco churn model. It is a very nice analysis and we thought that it would be interesting to compare the results to <a href="http://www.h2o.ai/">H2O</a>, which is a great tool for automated building of prediction models.<br />
<br />
Today Wit has published on Github his codes performing the analysis. You can check them out <a href="https://github.com/WLOGSolutions/telco-customer-churn-in-r-and-h2o">here</a>. The obtained model is pretty good, with AUC equal to 0.947 which is better than any of the models presented in Revolution blog. But, in my opinion the key value of the Wit's work is to show how simple it is to build models with H2O using R. From his <a href="https://github.com/WLOGSolutions/telco-customer-churn-in-r-and-h2o">Github repository</a> you can download a ready to run RStudio project and all the instructions needed to build the model in just a few lines of code.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4946490806848569840.post-50937214675907074762015-01-06T07:09:00.000-08:002015-01-06T07:09:32.766-08:00Sequence generation in R, Python and JuliaRecently I was comparing implementation of sequence generation functions in R, Python (numpy) and Julia. Interestingly even such basic functions have slight differences in implementation. In my opinion Julia provides the best solution and Python the worst.<br />
<br />
<a name='more'></a><br />
The test case will be a 6 element sequence from <span style="font-family: Courier New, Courier, monospace;">0.1</span> to <span style="font-family: Courier New, Courier, monospace;">1.1</span> namely <span style="font-family: Courier New, Courier, monospace;">[0.1, 0.3, 0.5, 0.7, 0.9, 1.1]</span>. In all languages we can generate it in three ways: by direct specification, by sequence giving its length and by sequence giving its step (there actually three flavors here: with exact and inexact right bound).<br />
<br />
The test codes follow but let me first give the conclusions:<br />
<br />
R:<div>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">seq</span> will generate the same sequence in all cases (even if <span style="font-family: Courier New, Courier, monospace;">to</span> parameter is a bit too small - see <span style="font-family: Courier New, Courier, monospace;">seq</span> help for details about 'just beyond', but then the last value in the sequence is corrected);</li>
<li>the sequence is not guaranteed to be equal to manually entered literals that represent the sequence most exactly;</li>
<li>if you generate the same sequence in the reversed order the result does not have to be equal.</li>
</ul>
Python:</div>
<div>
<ul>
<li>it matters if we generate sequence using <span style="font-family: Courier New, Courier, monospace;">linspace</span> or <span style="font-family: Courier New, Courier, monospace;">arange</span>;</li>
<li><span style="font-family: Courier New, Courier, monospace;">arange</span> excludes right end of the range specification; this actually can result in unexpected results; check <span style="font-family: Courier New, Courier, monospace;">numpy.arange(0.2, 0.6, 0.4)</span> vs <span style="font-family: Courier New, Courier, monospace;">numpy.arange(0.2, 1.6, 1.4)</span>;</li>
<li>the sequence is not guaranteed to be equal to manually entered literals that represent the sequence most exactly;</li>
<li>if you generate the same sequence in the reversed order the result does not have to be equal.</li>
</ul>
Julia:</div>
<div>
<ul>
<li>it may be important if we supply exact value of the last element of the sequence or it is inexact when we specify step (equivalent of <span style="font-family: Courier New, Courier, monospace;">by</span> in R), check out function <span style="font-family: Courier New, Courier, monospace;">colon</span> in <a href="https://github.com/JuliaLang/julia/blob/master/base/range.jl">range.jl</a>;</li>
<li>sequence generated by step and by number of elements can differ even if they have the same endpoints;</li>
<li>Julia actually is able to generate sequence equal to sequence that is equal to manually entered literals (sequence using step with exact right bound);</li>
<li>generating the sequence in the reversed order gives you the same result.</li>
</ul>
<br />
Below you can check the codes I used. Here are the results for R:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">x <- c(0.1, 0.3, 0.5, 0.7, 0.9, 1.1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y1 <- seq(from = 0.1, to = 1.1, length.out = 6)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y2 <- seq(from = 0.1, to = 1.1, by = 0.2)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y3 <- seq(from = 0.1, to = 1.2, by = 0.2)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y4 <- seq(from = 0.1, to = 1.099999999999, by = 0.2)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(x == y1)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE FALSE TRUE FALSE TRUE TRUE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(x == y2)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE FALSE TRUE FALSE TRUE TRUE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(y1 == y2)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE TRUE TRUE TRUE TRUE TRUE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(y2 == y3)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE TRUE TRUE TRUE TRUE TRUE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(y2 == y4)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE TRUE TRUE TRUE TRUE FALSE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">print(rev(seq(1.1,0.1, len=6)) == seq(0.1, 1.1, len=6))</span><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE TRUE TRUE TRUE FALSE TRUE</span><br />
<div>
<br /></div>
Now comes Python:<br />
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">import numpy</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">x = numpy.array([0.1, 0.3, 0.5, 0.7, 0.9, 1.1])</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y1 = numpy.linspace(0.1, 1.1, 6)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y2 = numpy.arange(0.1, 1.1, 0.2)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y3 = numpy.arange(0.1, 1.11, 0.2)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">print(x == y1)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [ True False True False True True]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">print(y2)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [ 0.1 0.3 0.5 0.7 0.9]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">print(x == y3)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [ True False False False False False]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">print(y1 == y3)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [ True True False True False False]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">print(list(reversed(numpy.linspace(1.1, 0.1, 6))) == y1)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [ True True True True False True]</span></div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
And finally Julia:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">x = [0.1, 0.3, 0.5, 0.7, 0.9, 1.1]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y1 = linspace(0.1, 1.1, 6)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y2 = [0.1:0.2:1.1]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y3 = [0.1:0.2:nextfloat(1.1)]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">y4 = [0.1:0.2:prevfloat(1.1)]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(x .== y1)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Bool[true,false,true,false,false,true]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(x .== y2)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Bool[true,true,true,true,true,true]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(y1 .== y3)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Bool[true,true,true,true,false,true]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(y2 .== y3)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Bool[true,false,true,false,true,true]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(y4)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># [0.1,0.30000000000000004,0.5,0.7000000000000001,0.9]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">println(linspace(0.1, 1.1, 6) .== reverse(linspace(1.1, 0.1, 6)))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Bool[true,true,true,true,true,true]</span></div>
</div>
</div>
<div>
<br /></div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-4946490806848569840.post-19326392987207739632014-07-16T11:13:00.000-07:002014-07-18T05:03:32.645-07:00Comparing localsolver with Rglpk on k-medoids exampleRecently I have coauthored a new <a href="http://cran.r-project.org/web/packages/localsolver/index.html" target="_blank">localsolver</a> package that can be used to solve large scale optimization problems from R. It is a wrapper around <a href="http://www.localsolver.com/" target="_blank">commercial solver</a> that is free for academia. If you are interested why it is worthwhile to give it a look - read on.<br />
<div>
<a name='more'></a>There is a host of packages on optimization <a href="http://cran.r-project.org/web/views/Optimization.html" target="_blank">CRAN Task View</a> so why bother with a new one? There are two reasons: (1) flexibility and speed of LocalSolver optimization engine and (2) ease of use - which is often a problem with traditional packages like Rglpk - especially for large scale problems.</div>
<div>
Today I will introduce the package and focus on ease of use. Next time I will provide some performance comparisons to existing solvers.</div>
<div>
<br /></div>
<h3>
Before we begin</h3>
<div>
First you need to setup your environment. There are two steps.</div>
<div>
You should start with <a href="http://www.localsolver.com/download.html" target="_blank">downloading LocalSolver</a> and installing it.</div>
<div>
Next simply install <a href="http://cran.r-project.org/web/packages/localsolver/index.html" target="_blank">localsolver</a> package (if you have problems with installation on Mac use <span style="font-family: Courier New, Courier, monospace;">type="source"</span> option of <span style="font-family: Courier New, Courier, monospace;">install.pacakges</span> should solve them).</div>
<div>
In case LocalSolver is not in search path of your system after installation either add it or remember its path as in such cases you can pass it to computational routines.</div>
<div>
Now we are ready to go.</div>
<div>
<br /></div>
<h3>
The test problem</h3>
<div>
As this is an example of ease of use we will focus on an elementary integer programming task. Suppose that we have 20 locations. Out of them we have to select 5 depots and we want to assign every location to nearest depot. We know distances between all stores (distance matrix does not have to be symmetric). Our objective is to minimize sum of all distances between depots and locations assigned to them. We can see that the problem is simply a <a href="http://en.wikipedia.org/wiki/K-medoids" target="_blank">k-medoids clustering</a> task but without assumption of symmetry of distance matrix.<br />
<br />
Let us start with code that generates test matrices for our examples:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">kmedoids.generate.data <- function(seed, N, P) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if(!is.null(seed)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> set.seed(seed) </span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> InfiniteDist <- 10000.0</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Dist <- matrix(round(runif(n=N * N) * 100, digits = 0),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> nrow = N)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> diag(Dist) <- 0</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> return(list(</span><br />
<span style="font-family: Courier New, Courier, monospace;"> N = as.integer(N),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> P = as.integer(P), </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Dist = Dist,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> InfiniteDist = InfiniteDist))</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">data <- kmedoids.generate.data(seed = 1234, N = 20L, P = 5L)</span><br />
<div>
<br /></div>
One thing that is worth explaining is that in the code we define <span style="font-family: 'Courier New', Courier, monospace;">InfiniteDist</span> which will serve us as surrogate for infinity (all actual distances are shorter than it).<br />
<br />
Now let me present how the problem can be solved with localsolver and Rglpk.<br />
<br />
<h3>
localsolver approach</h3>
<div>
Here is the code:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">library(localsolver)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">lsp.model <- "function model() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x[1..N] <- bool() ; // point i is in P iff x[i] = 1</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> constraint sum[i in 1..N](x[i]) == P ;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> minDist[i in 1..N] <- min[j in 1..N](</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x[j] ? Dist[j][i] : InfiniteDist);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">// minimize sum of distances</span><br />
<span style="font-family: Courier New, Courier, monospace;"> objective <- sum[i in 1..N]( minDist[i] ) ;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> minimize objective;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">lsp <- ls.problem(model.text.lsp = lsp.model)</span><br />
<span style="font-family: Courier New, Courier, monospace;">lsp <- set.params(lsp = lsp, lsTimeLimit = 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">lsp <- set.params(lsp = lsp, lsNbThreads = 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">lsp <- add.output.expr(lsp=lsp, expr.text.lsp = "x",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> dimensions = data$N)</span><br />
<span style="font-family: Courier New, Courier, monospace;">lsp <- add.output.expr(lsp=lsp, expr.text.lsp = "objective",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> dimensions = 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">lsp.solution <- ls.solve(lsp = lsp, data = data)</span><br />
<div>
<br /></div>
Notice that we can define the optimization problem using a simple domain specific language as a string. Interestingly in the string we can use all variables defined in <span style="font-family: 'Courier New', Courier, monospace;">data</span> list as localsolver is able to interact with extrernal data source. Additionally using <span style="font-family: 'Courier New', Courier, monospace;">set.params</span> and <span style="font-family: 'Courier New', Courier, monospace;">add.output.expr</span> we can set up the parameters of optimization engine and optimization output we will want to collect.<br />
As a result we get a list with optimization output. Importantly the contents of the list can be controlled programatically as mentioned above.<br />
<br />
<h3>
Rglpk approach</h3>
<div>
Now let us solve the same problem using Rglpk:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">library(Rglpk)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">objective <- c(as.numeric(data$Dist), rep(0, data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">types <- c(rep("C", length(data$Dist)),</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rep("B", data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Exactly P centroids</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># sum[i in 1..N] y[i] == P</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">mat <- matrix(data = c(rep(0, length(data$Dist)),</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rep(1, data$N)), nrow=1)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">rhs <- c(data$P)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">dir <- c("==")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Each datapoint only to one centroid</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># x[i,j] - ith datapoint to jth centroid</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># For each j in 1..N sum[i in 1..N] x[i,j] == 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">for(j in 1:data$N) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> mat <- rbind(mat,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> c(</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rep(0, (j-1)*data$N),</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rep(1, data$N),</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rep(0, ncol(mat) - j*data$N))) </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">rhs <- c(rhs, rep(1, data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">dir <- c(dir, rep("==", data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Client cannot be assigned to notselected centroid</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># x[i,j] <= y[j]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># x[i,j] - y[j] <= 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">for(i in 1:data$N) { </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for(j in 1:data$N) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> row <- rep(0, ncol(mat)) </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> row[(i-1)*data$N + j] <- 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> row[length(data$Dist) + j] <- -1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> mat <- rbind(mat, row) </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">rhs <- c(rhs, rep(0, data$N*data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">dir <- c(dir, rep("<=", data$N*data$N))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">glpk.sol <- Rglpk_solve_LP(</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> obj = objective,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> mat = mat,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> dir = dir,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rhs = rhs,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> types = types,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> max = FALSE,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> control = list(verbose = TRUE,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> canonicalize_status = TRUE))</span></div>
<div>
<br /></div>
</div>
In my opinion the code looks cryptic in comparison to localsolver specification. There are two reasons for such a situation:<br />
<br />
<ol>
<li>we have to have two mix two types of decision variables: location of depots and assignment of locations to depots</li>
<li>we have to construct a complex constraints matrix combining different types of conditions in one big data structure</li>
</ol>
<div>
In fact even in order to directly learn which locations are selected as depots we have to dig into Rglpk output with <span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;"> </span><span style="background-color: white; color: #222222; font-size: 13px;"><span style="font-family: Courier New, Courier, monospace;">tail(glpk.sol$solution, n = data$N)</span></span>.</div>
<br />
<h3>
Summary</h3>
<div>
We have tested localsolver on linear integer programming problem. I wanted to show that the package allows the user to solve optimization tasks:<br />
<br />
<ul>
<li>using natural (mathematical) formulation of objective function and constraints</li>
<li>allowing to dynamically update reference data: that <span style="font-family: 'Courier New', Courier, monospace;">lsp.model</span> string did not have any hard coded constants - everything is loaded from metadata provided as R list.</li>
</ul>
<div>
In my opinion it is not only more elastic in comparison to Rglpk solution presented above but also more robust. Coding Rglpk solution took much more time as it required much more thought in the design of <span style="font-family: 'Courier New', Courier, monospace;">Rglpk_solve_LP</span> input variables and a lot of corrections of errors in the code.</div>
<div>
<br /></div>
<div>
What is more localsolver is not only more user friendly but is also not limited to mixed integer linear optimization - it can solve highly complex non-linear tasks as well.</div>
<div>
<br /></div>
<div>
Now a natural question arises: what is the performance of localsolver in terms of speed and the quality of obtained solution. I will tackle them in my next post.</div>
</div>
</div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-51203381679445256712014-06-25T06:03:00.000-07:002014-06-25T06:04:03.428-07:00R Scrabble: Part 2Ivan Nazarov and Bartek Chroł gave very interesting comments to <a href="http://rsnippets.blogspot.ca/2014/06/rgolf-ngsl-scrabble.html" target="_blank">my last post</a> on counting number of subwords in NGSL words. In particular they proposed large speedups of my code. So I thought to try checking a larger data set. So today I will work with <a href="http://en.wikipedia.org/wiki/Official_Tournament_and_Club_Word_List" target="_blank">TWL2006</a> - the official word authority for tournament Scrabble in the USA.<br />
The question is whether the exponential relationship between the number of letters in the word and the number of its subwords that is observed in NGSL data set still holds for TWL2006.<br />
<a name='more'></a>The challenge is that NGSL has 2801 words and TWL2006 is much larger with 178691 words. You can download the file <a href="http://bogumilkaminski.pl/share/TWL2006.txt" target="_blank">TWL2006.txt</a> which contains the words and was prepared (converted to lowercase and sorted by word length) using data from <a href="http://www.isc.ro/en/commands/lists.html" target="_blank">Internet Scrabble Club</a> website.<br />
<br />
You could run codes from comments to my last post to obtain the results, but it takes ages to compute (over 1 hour). Therefore I have written the data preparation step procedures in Java - which reduced the time needed to perform the analysis down to around 1 minute.<br />
<br />
So first let us start with Java code that computes number of subwords for each word in TSL2006 dictionary:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">package scrabble;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">import java.io.*;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">public class Scrabble implements Runnable {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public static int[][] b = new int[178691][26];</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public static int wlen[] = new int[178691];</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // Set number of threads to match your hardware</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public static final int MAX_THREADS = 6;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public static void main(String[] args)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> throws FileNotFoundException, IOException {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String file = "TWL2006.txt";</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int i;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> try (BufferedReader r =</span><br />
<span style="font-family: Courier New, Courier, monospace;"> new BufferedReader(new FileReader(file))) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> i = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (String line; (line = r.readLine()) != null; i++) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (char c : line.toCharArray()) { b[i][c - 'a']++; }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> wlen[i] = line.length();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (i = 0; iI < MAX_THREADS; i++) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> new Thread(new Scrabble(i)).start();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> private final int counter;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> public Scrabble(int counter) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> this.counter = counter;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> @Override</span><br />
<span style="font-family: Courier New, Courier, monospace;"> public void run() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String filename = "result_" + counter + ".txt";</span><br />
<span style="font-family: Courier New, Courier, monospace;"> try (PrintWriter w = new PrintWriter(filename)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> w.println("length, subwords");</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (int i = counter; i < b.length; i += MAX_THREADS) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int[] base = b[i];</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int subwordcount = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (int j = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> (j < b.length) && (wlen[j] <= wlen[i]); j++) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int[] subword = b[j];</span><br />
<span style="font-family: Courier New, Courier, monospace;"> boolean issubword = true;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (int k = 0; k < 26; k++) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (subword[k] > base[k]) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> issubword = false; break;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (issubword) { subwordcount++; }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> w.println(wlen[i] + ", " + subwordcount);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (FileNotFoundException ex) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> System.out.println("Failed to open file " + filename);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } </span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<div>
<br /></div>
The code uses idea proposed in comments to my last post to hold a table counting number of occurrences of each letter in each word (array <span style="font-family: Courier New, Courier, monospace;">b</span>). In order to speed up the computations the code takes advantage from the fact that words in file TWL2006.txt are sorted by their length and additionally is parallelized (6 threads by default) so produces 6 files named <span style="font-family: Courier New, Courier, monospace;">result_[i].txt</span>., where <span style="font-family: Courier New, Courier, monospace;">i</span> changes from 0 to 6. If you do not have Java compiler you can download <a href="http://bogumilkaminski.pl/share/Scrabble.jar" target="_blank">Scrabble.jar here</a> (but it is better to build the jar file from the source as your browser might complain about downloading jar files from untrusted sources).<br />
<br />
And now the code that runs the analysis. It comes in three parts. First we plot the relationship between word length and logarithm of mean subword count in TWL2006 data set:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">library(dplyr)</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"># you may run it outside R and skip the Java code</span><br />
<span style="font-family: Courier New, Courier, monospace;"># computations should take ~1 minute for 6 threads</span><br />
<span style="font-family: Courier New, Courier, monospace;"># additionally safeguard against accidental multiple generation</span><br />
<span style="font-family: Courier New, Courier, monospace;">if (length(list.files(pattern="result_\\d+\\.txt")) == 0) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> system("java -jar Scrabble.jar")</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">results <- list.files(pattern="result_\\d+\\.txt")</span><br />
<span style="font-family: Courier New, Courier, monospace;">rbind_all(lapply(results, read.csv)) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> group_by(length) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> summarise(subwords=log(mean(subwords))) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> plot(xlab = "word length", ylab = "log(mean(subwords))")</span><br />
<br />
In the process we can see the power of <span style="font-family: Courier New, Courier, monospace;">dplyr</span> ;). The result is the following:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-0UQrzrdPF60/U6rDYDVPM1I/AAAAAAAAAZk/YAKncF4AvG8/s1600/len_subword.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-0UQrzrdPF60/U6rDYDVPM1I/AAAAAAAAAZk/YAKncF4AvG8/s1600/len_subword.png" height="356" width="400" /></a></div>
And we get a different result. Number of subwords grows subexponentially with word length.<br />
<br />
In order to examine the reason for such a situation let us plot the word length distributions of NGSL and TWL datasets:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">twl2006_l <- nchar(readLines("TWL2006.txt"))</span><br />
<span style="font-family: Courier New, Courier, monospace;">ngsl101_l <- nchar(readLines("NGSL101.txt")[-1])</span><br />
<span style="font-family: Courier New, Courier, monospace;">par(lend=2)</span><br />
<span style="font-family: Courier New, Courier, monospace;">hist(ngsl101_l, col=rgb(1,0,0,0.5), 0:15, FALSE, ylim=c(0, 0.2),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> xlab="letter count", main="")</span><br />
<span style="font-family: Courier New, Courier, monospace;">hist(twl2006_l, col=rgb(0,0,1,0.5), 0:15, FALSE, add=TRUE)</span><br />
<span style="font-family: Courier New, Courier, monospace;">legend("topright", c("NGSL101", "TWL2006"), title = "Dictionary",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> col=c(rgb(1,0,0,0.5), rgb(0,0,1,0.5)), lty=1, lwd=14)</span><br />
<div>
<br /></div>
The histograms produced are the following:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-QNe1CuFKKtM/U6rESLh3HYI/AAAAAAAAAZs/xQC3Wdr-yxQ/s1600/nchar_hist.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-QNe1CuFKKtM/U6rESLh3HYI/AAAAAAAAAZs/xQC3Wdr-yxQ/s1600/nchar_hist.png" height="365" width="400" /></a></div>
So we can see that basic (NGSL) words tend to be shorter than TWL words. We can then reason that it is easier for a sort word to be a subword of some longer word.<br />
<br />
Having established the difference in word length distribution another question came to me. Does the distribution of letters in words of a given length differ significantly between NGSL and TWL. Here is the code that checks this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">letter.freq <- function(file.name, skip){</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sdf <- read.table(file.name, as.is = T,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> col.names = "word", skip = skip) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> mutate(len = nchar(word)) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # strip to words from 2 to 14 letters for comparability</span><br />
<span style="font-family: Courier New, Courier, monospace;"> filter(len > 1 & len < 15) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> group_by(len) %>%</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # generate distribution of letters by word length</span><br />
<span style="font-family: Courier New, Courier, monospace;"> do(props=prop.table(table(factor(</span><br />
<span style="font-family: Courier New, Courier, monospace;"> strsplit(paste(.$word, collapse=""),"")[[1]],letters))))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> letfreq <- do.call(cbind, sdf$props)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> colnames(letfreq) <- 2:14</span><br />
<span style="font-family: Courier New, Courier, monospace;"> letfreq</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">library(lattice)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">twl2006 <- letter.freq("TWL2006.txt", 0)</span><br />
<span style="font-family: Courier New, Courier, monospace;">ngsl101 <- letter.freq("NGSL101.txt", 0)</span><br />
<span style="font-family: Courier New, Courier, monospace;">levelplot(twl2006-ngsl101, main = "TWL2006-NGSL101",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> xlab = "letter", ylab = "word length")</span><br />
<div>
<br /></div>
The plot shows the differences of the distributions:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-0vA0HfBfERI/U6rFhDx8xvI/AAAAAAAAAZ4/dZ5GDfqHyII/s1600/freq_comp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-0vA0HfBfERI/U6rFhDx8xvI/AAAAAAAAAZ4/dZ5GDfqHyII/s1600/freq_comp.png" height="242" width="400" /></a></div>
Interestingly we again can notice significant differences. For example the letter "t" seems to be on-the-average more frequent in NGSL words and the letter "s" in TWL words.<br />
<br />
Unfortunately I am not an English language specialist to try to explain the patterns presented on the plots I have prepared today. If you have any insights concerning the distribution of letters in English words I welcome you to share your thoughts (and codes :)) in comments.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-40446599219607025522014-06-14T02:55:00.001-07:002014-06-25T05:04:25.052-07:00RGolf: NGSL ScrabbleIt is last part of RGolf before summer. As R excels in visualization capabilities today the task will be to generate a plot.<br />
<br />
<a name='more'></a>We will work with <a href="http://www.newgeneralservicelist.org/">NGSL</a> data - a list of 2801 important vocabulary words for students of English as a second language. I have prepared the list as a <a href="http://bogumilkaminski.pl/share/NGSL101.txt" target="_blank"><span style="font-family: Courier New, Courier, monospace;">NGSL101.txt</span> file for download</a>.<br />
<div>
<br />
Let us move to the task. Load NGSL101.txt into R. For each word in the list we want to calculate the number of other words from the list that can be arranged from a subset of letters from the original words (like in scrabble). For example for word "shoot" we have the following list of its subwords: "to", "so", "too", "hot", "shoot", "host" and "shot". As a product we want to plot the relationship between number of letters in the word and the logarithm of number of its subwords.</div>
<div>
<br /></div>
<div>
The rules of the game are standard:</div>
<div>
<div>
(1) generate the plot in as few keystrokes as possible;</div>
<div>
(2) plot formatting (e.g. x and y axis titles, type of plot) is not important;</div>
<div>
(3) one line of code may not be longer than 80 characters;</div>
<div>
(4) the solution must be in base R only (no package loading is allowed);<br />
(5) assume that you have NGSL101.txt file in your R working directory.</div>
</div>
<div>
As always - if you have a nice solution please submit a comment :).<br />
<br />
Warning! This time the task takes a bit more time to compute so it is worth to do the development and testing of the solution on the subset of NGSL word list.<br />
<br /></div>
<div>
Here is my attempt in 169 keystrokes:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">d=scan("NGSL101.txt","",skip=1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">a=s((s=sapply)(strsplit(d,""),sort),paste,collapse=".*")</span><br />
<span style="font-family: Courier New, Courier, monospace;">y=log(s(a,function(z)sum(s(a,function(i)grepl(i,z)))))</span><br />
<span style="font-family: Courier New, Courier, monospace;">plot(by(y,nchar(d),mean))</span><br />
<br />
And the output is the following:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/--ts7PyuZe6s/U5wZWht0v6I/AAAAAAAAAY8/E_XgdORPxw8/s1600/output.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/--ts7PyuZe6s/U5wZWht0v6I/AAAAAAAAAY8/E_XgdORPxw8/s1600/output.png" height="319" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
As we can see the number of subwords approximately on the average grows exponentially with the number of letters in a word.<br />
<br />
And here is a verbose version of the solution with comments (warning again - it is slower than the solution given above):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">d <- readLines("NGSL101.txt")</span><br />
<span style="font-family: Courier New, Courier, monospace;">d <- d[-1] # remove first line from the dataset as it is a comment</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">is.subword <- function(test, ref) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # we check if test is a subword of ref by applying </span><span style="font-family: 'Courier New', Courier, monospace;">regular</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # expression on sorted letters contained in both words</span><br />
<span style="font-family: Courier New, Courier, monospace;"> test <- paste(sort(strsplit(test, "")[[1]]),collapse=".*")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ref <- paste(sort(strsplit(ref, "")[[1]]),collapse="")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # grepl returns true is match is found</span><br />
<span style="font-family: Courier New, Courier, monospace;"> grepl(test, ref)</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># traverse all words in d and count number of matches</span><br />
<span style="font-family: Courier New, Courier, monospace;">count.subwords <- function(ref) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sum(sapply(d, is.subword, ref = ref))</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">x2 <- nchar(d)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y2 <- log(sapply(d, count.subwords))</span><br />
<span style="font-family: Courier New, Courier, monospace;">y2.means <- tapply(y2, x2, mean)</span><br />
<span style="font-family: Courier New, Courier, monospace;">plot(y2.means)</span><br />
<div>
<br /></div>
</div>
Unknownnoreply@blogger.com7tag:blogger.com,1999:blog-4946490806848569840.post-46108897921891888232014-05-30T04:39:00.000-07:002014-05-30T04:39:22.322-07:00RGolf: rolling windowI have learned a lot from <a href="http://rsnippets.blogspot.com/2014/05/rgolf.html">my last RGolf post</a>. Therefore today I have another problem from practice.<br />
<br />
<a name='more'></a>You have a data set on values of contracts signed by ten salesmen. It has three columns: person id (<span style="font-family: Courier New, Courier, monospace;">p</span>), contract value (<span style="font-family: Courier New, Courier, monospace;">v</span>) and time (<span style="font-family: Courier New, Courier, monospace;">t</span>).<br />
Here is the code that generates the test data set.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">n <- 4000</span><br />
<span style="font-family: Courier New, Courier, monospace;">y <- 10</span><br />
<span style="font-family: Courier New, Courier, monospace;">set.seed(1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">d <- data.frame(p = sample(letters[1:y], n, rep=T),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> v = runif(n),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> t = sample(seq(as.Date("2000-1-1"),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> by="day", len=n)))</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">head(d)</span><br />
<span style="font-family: Courier New, Courier, monospace;"># p v t</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 1 c 0.18776846 2001-05-28</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 2 d 0.50475902 2001-05-25</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 3 f 0.02728685 2008-07-06</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 4 j 0.49629785 2004-08-06</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 5 c 0.94735171 2007-10-31</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 6 i 0.38118213 2001-09-04</span><br />
<br />
For each salesmen we want to find the 90-day period in which she generated the highest sum of sales.<br />
The task is to generate a data frame that has three columns: person id (<span style="font-family: Courier New, Courier, monospace;">p</span>), maximal sum of sales (<span style="font-family: Courier New, Courier, monospace;">v</span>), start of the 90-day period (<span style="font-family: Courier New, Courier, monospace;">t</span>) and save it to a variable named <span style="font-family: Courier New, Courier, monospace;">r</span>.<br />
In situations where there are multiple such 90-day periods you are to report the date of the earliest contract that was signed during such a period.<br />
<br />
The rules of engagement are exactly as last time:<br />
(1) generate data frame r as few keystrokes as possible,<br />
(2) one line of code may not be longer than 80 characters,<br />
(3) the solution must be in base R only (no package loading is allowed).<br />
<br />
Here is my attempt consisting of 225 characters:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">x=max(d$t);m=matrix(0,x+89,y)</span><br />
<span style="font-family: Courier New, Courier, monospace;">m[cbind(d$t, d$p)]=d$v</span><br />
<span style="font-family: Courier New, Courier, monospace;">s=sapply(1:x,function(z)</span><br />
<span style="font-family: Courier New, Courier, monospace;">colSums(m[z+0:89,]))*t(m>0)[,1:x]</span><br />
<span style="font-family: Courier New, Courier, monospace;">u=apply(s,1,function(z)c(max(z),which.max(z)))</span><br />
<span style="font-family: Courier New, Courier, monospace;">r=data.frame(p=letters[1:y],v=u[1,],t=u[2,]+as.Date("1970-01-01"))</span><br />
<br />
It produces the following output:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">r</span><br />
<span style="font-family: Courier New, Courier, monospace;"># p v t</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 1 a 10.198044 2009-10-11</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 2 b 9.265335 2002-08-28</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 3 c 9.735401 2008-02-21</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 4 d 9.479942 2004-09-07</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 5 e 11.036041 2010-09-16</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 6 f 10.602446 2002-04-04</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 7 g 9.927153 2007-08-11</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 8 h 10.917856 2007-04-26</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 9 i 10.027341 2008-10-26</span><br />
<span style="font-family: Courier New, Courier, monospace;"># 10 j 9.965738 2004-12-28</span><br />
<br />
Here is a more verbose version that is easier to read:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">d_o <- d[order(d$t),] # order d by time</span><br />
<span style="font-family: Courier New, Courier, monospace;">testr <- NULL</span><br />
<span style="font-family: Courier New, Courier, monospace;">for (p in levels(d_o$p)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> d_o_p <- d_o[d_o$p == p,] # subset ordered d by player</span><br />
<span style="font-family: Courier New, Courier, monospace;"> best_v_sum <- -Inf</span><br />
<span style="font-family: Courier New, Courier, monospace;"> best_i <- NA</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for (i in 1:nrow(d_o_p)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> d_o_p_t <- d_o_p[d_o_p$t < d_o_p$t[i] + 90 &</span><br />
<span style="font-family: Courier New, Courier, monospace;"> d_o_p$t >= d_o_p$t[i],]</span><br />
<span style="font-family: Courier New, Courier, monospace;"> # subset ordered d by player and rolling window</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (sum(d_o_p_t$v) > best_v_sum) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> best_v_sum <- sum(d_o_p_t$v)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> best_i <- i</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> testr <- rbind(testr, data.frame(p = p, v = best_v_sum,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> t = d_o_p$t[best_i]))</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">all.equal(r,testr)</span><br />
<!-----><!-----><br />
<span style="font-family: Courier New, Courier, monospace;"># [1] TRUE</span><br />
<div>
<br /></div>
<div>
Any competing solution should pass <span style="font-family: Courier New, Courier, monospace;">all.equal</span> test.</div>
<div>
<br /></div>
<div>
As last time - please post your solutions in comments to the post.</div>
Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-4946490806848569840.post-78481240279115981542014-05-16T12:30:00.003-07:002014-05-16T12:43:39.949-07:00RGolfIts time for some fun today - because its Friday as <a href="http://profile.typepad.com/revolutiondavid">David Smith</a> says :).<br />
<br />
There are many <a href="http://en.wikipedia.org/wiki/Code_golf">code golf</a> sites, even some support R. However, most of them are algorithm oriented. A true RGolf competition should involve transforming a source data frame to some target format data frame.<br />
<br />
So the challenge today will be to write a shortest code in R that performs a required data transformation.<br />
<br />
<a name='more'></a>Let's start with the data transformation task (actually the problem was taken from a real data set I have recently analyzed).<br />
<br />
We are running a survey. Each respondent is asked some subset of possible questions (labelled by letters) and answers the question positively (1) or negatively (0). As input we are given a data frame with two columns: labels of questions asked (as letters) and sequence of answers given to them (string of 0's and 1's). A good R example is better than 1000 words :):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">> set.seed(1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">> questions <- replicate(1000, paste(sample(letters[1:10],</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sample.int(4) + 2), collapse = ""))</span><br />
<span style="font-family: Courier New, Courier, monospace;">> answers <- sapply(questions, function(x) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> paste(as.character(rbinom(nchar(x), 1, 0.5)),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> collapse = "") })</span><br />
<span style="font-family: Courier New, Courier, monospace;">> dataset <- data.frame(questions, answers,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> stringsAsFactors = FALSE)</span><br />
<span style="font-family: Courier New, Courier, monospace;">> head(dataset)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> questions answers</span><br />
<span style="font-family: Courier New, Courier, monospace;">1 cihe 1100</span><br />
<span style="font-family: Courier New, Courier, monospace;">2 gdjie 01100</span><br />
<span style="font-family: Courier New, Courier, monospace;">3 cfbhja 001000</span><br />
<span style="font-family: Courier New, Courier, monospace;">4 febj 1110</span><br />
<span style="font-family: Courier New, Courier, monospace;">5 ehfid 01101</span><br />
<span style="font-family: Courier New, Courier, monospace;">6 hgdic 10010</span><br />
<div>
<br /></div>
We will want to transform <span style="font-family: Courier New, Courier, monospace;">dataset</span> to the following wide format (stored in <span style="font-family: Courier New, Courier, monospace;">dataset2</span>):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">> head(dataset2)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> a b c d e f g h i j</span><br />
<span style="font-family: Courier New, Courier, monospace;">1 NA NA 1 NA 0 NA NA 0 1 NA</span><br />
<span style="font-family: Courier New, Courier, monospace;">2 NA NA NA 1 0 NA 0 NA 0 1</span><br />
<span style="font-family: Courier New, Courier, monospace;">3 0 1 0 NA NA 0 NA 0 NA 0</span><br />
<span style="font-family: Courier New, Courier, monospace;">4 NA 1 NA NA 1 1 NA NA NA 0</span><br />
<span style="font-family: Courier New, Courier, monospace;">5 NA NA NA 1 0 1 NA 1 0 NA</span><br />
<span style="font-family: Courier New, Courier, monospace;">6 NA NA 0 0 NA NA 0 1 1 NA</span><br />
<div>
<br /></div>
<div>
The challenge is to transform <span style="font-family: Courier New, Courier, monospace;">dataset</span> in such a way to generate <span style="font-family: Courier New, Courier, monospace;">dataset2</span> in as few keystrokes as possible, assuming that number of questions and number of respondents (respectively equal to 10 and 1000 in example data set) is unknown. The constraints are that one line of code may not be longer than 80 characters and the solution must be in base R only (no package loading is allowed).</div>
<div>
<br /></div>
<div>
Here is my attempt:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">d<-dataset;y<-sort(unique(strsplit(paste(d[[1]],collapse=""),"")[[1]]))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">d2<-data.frame(t(mapply(function(q,a){r<-rep(NA,length(y))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">r[grepl(paste("[",q,"]",sep=""),y)]<-as.numeric(strsplit(a,split="")[[1]][</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">order(strsplit(q,split="")[[1]])]);names(r)<-y;r},d[[1]],d[[2]],USE.NAMES=F)))</span></div>
</div>
<div>
<br /></div>
<div>
It has 284 characters (including 3 newline characters). If you take the challenge and have a shorter solution that produces exactly the same <span style="font-family: Courier New, Courier, monospace;">dataset2</span> data set for a given input post a comment ;). In order for the comment to be accepted the solution must be robust to changes of generated data set (different number of possible questions and answers).</div>
<div>
<br /></div>
<div>
Before I quit I present the same code in slightly more readable format and commented:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"># extract all classes that exist in dataset$questions</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"># and sort them</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">classes <- sort(unique(strsplit(paste(dataset$questions,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> collapse = ""), "")[[1]]))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"># change one pair of questions and answers into</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"># a full vector containing all classes sorted</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">process.qa <- function(q, a) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> res <- rep(NA, length(classes)) # initially no classes are set</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> qs <- strsplit(q, split="")[[1]] # extract question classes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> # extract answers and sort them in order of question classes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> as <- as.numeric(strsplit(a, split="")[[1]][order(qs)])</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> # update result with answers for existing questions</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> res[grepl(paste("[",q, "]", sep=""), classes)] <- as</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> names(res) <- classes</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> res</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;">dataset2 <- data.frame(t(mapply(process.qa,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: 9pt;"> dataset$questions, dataset$answers, USE.NAMES = F)))</span></div>
</div>
<div>
<br /></div>Unknownnoreply@blogger.com11tag:blogger.com,1999:blog-4946490806848569840.post-18940930038159010152014-01-26T03:20:00.000-08:002014-01-26T03:20:05.672-08:00Tuning optim with parscaleI often get questions what is the use of <span style="font-family: Courier New, Courier, monospace;">parscale</span> parameter in <span style="font-family: Courier New, Courier, monospace;">optim</span> procedure in GNU R. Therefore I have decided to write a simple example showing its usage and importance.
<br />
<a name='more'></a><br />
The function I test is a simplified version of estimation problem I had to solve recently. We have two explanatory variables <span style="font-family: Courier New, Courier, monospace;">x1</span> and <span style="font-family: Courier New, Courier, monospace;">x2</span>, The issue is that <span style="font-family: Courier New, Courier, monospace;">x1</span> range is much smaller than <span style="font-family: Courier New, Courier, monospace;">x2</span>. In the code below I control it by <span style="font-family: Courier New, Courier, monospace;">ratio</span> parameter (set to 200 here). Next we have <span style="font-family: Courier New, Courier, monospace;">y = exp(ratio*x1)+x2</span>. This is an explained variable - for simplicity we assume that there is no error in the model.<br />
<br />
We take 1000 observations and want to fit the regression by minimizing sum of squared errors. I find the parameters using <span style="font-family: Courier New, Courier, monospace;">optim</span> function using Nelder-Mead, BFGS and conjugate gradient methods. In the procedure <span style="font-family: Courier New, Courier, monospace;">maxit</span> is set to 200000 to ensure that optimization converges (for default <span style="font-family: Courier New, Courier, monospace;">maxit</span> value we would not even get algorithm to converge in some of the cases considered).<br />
<br />
The final thing is <span style="font-family: Courier New, Courier, monospace;">parscale</span>. We test two values - in the first case we leave it unmodified. In this case all variables are treated equally (although we know that unit change of <span style="font-family: Courier New, Courier, monospace;">x1</span> and <span style="font-family: Courier New, Courier, monospace;">x2</span> has significantly different influence on the objective function). In the second case we hint <span style="font-family: Courier New, Courier, monospace;">optim</span> to rescale <span style="font-family: Courier New, Courier, monospace;">x1</span> by <span style="font-family: Courier New, Courier, monospace;">ratio</span>. Now unit change of rescaled <span style="font-family: Courier New, Courier, monospace;">x1</span> and <span style="font-family: Courier New, Courier, monospace;">x2</span> have approximately equal influence on the objective function. We want to compare number of function and gradient evaluations in all 6 considered scenarios.<br />
<br />
Here is the code that does all the calculations. The printed output is given in the comment at the end of the code:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">f <- function(a) { exp(a[1] * x1) + a[2] * x2 }</span><br />
<span style="font-family: Courier New, Courier, monospace;">error <- function(a) {sum((y - f(a)) ^ 2) }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">set.seed(1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">n <- 1000</span><br />
<span style="font-family: Courier New, Courier, monospace;">ratio <- 200</span><br />
<span style="font-family: Courier New, Courier, monospace;">a <- c(ratio, 1)</span><br />
<span style="font-family: Courier New, Courier, monospace;">x1 <- runif(1000) / ratio</span><br />
<span style="font-family: Courier New, Courier, monospace;">x2 <- runif(1000)</span><br />
<span style="font-family: Courier New, Courier, monospace;">y <- f(a)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for (m in c("Nelder-Mead", "BFGS", "CG")) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> result1 <- optim(rep(1, 2), error, method = m,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> control = list(maxit=200000))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> result2 <- optim(rep(1, 2), error, method = m,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> control = list(maxit=200000, parscale = a))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat(format(m, justify="left", width=14), names(result1$count),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "\n No parscale:", format(result1$counts, width=8),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "\n Parscale: ", format(result2$counts, width=8),</span><br />
<span style="font-family: Courier New, Courier, monospace;"> "\n\n")</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"># Nelder-Mead function gradient</span><br />
<span style="font-family: Courier New, Courier, monospace;"># No parscale: 17703 NA</span><br />
<span style="font-family: Courier New, Courier, monospace;"># Parscale: 63 NA</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># BFGS function gradient</span><br />
<span style="font-family: Courier New, Courier, monospace;"># No parscale: 90 42</span><br />
<span style="font-family: Courier New, Courier, monospace;"># Parscale: 65 16</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># CG function gradient</span><br />
<span style="font-family: Courier New, Courier, monospace;"># No parscale: 448479 112123</span><br />
<span style="font-family: Courier New, Courier, monospace;"># Parscale: 341 69</span><br />
<br />
It can be clearly seen that appropriate rescaling of parameters has very significant influence on optimization converegence speed. It is especially important in Nelder-Mead and conjugate gradient methods in our example.Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-4946490806848569840.post-33229648766022572282013-12-28T06:34:00.000-08:002013-12-28T06:34:32.404-08:00GNU R vs Julia: is it only a matter of devectorization?Recently I have read a post on <a href="http://www.johnmyleswhite.com/notebook/2013/12/22/the-relationship-between-vectorized-and-devectorized-code/">benefits of code devectorization</a> in Julia. The examples given there inspired me to perform my own devectorization exercise. I decided to use bootstrapping as a test ground. The results are quite interesting (and not so bad for GNU R).<br />
<br />
<a name='more'></a>The task is very simple (and typical):<br />
<br />
<ol>
<li>generate 10000 elements sample from uniform distribution</li>
<li>1000 times perform bootstrap sample of the vector and calculate its standard deviation</li>
<li>return standard deviation of the bootstrap distribution</li>
<li>Perform steps 1-3 four times and record computation time</li>
</ol>
<br />
Let us start with GNU R implementation:<br />
<span style="font-family: Courier New, Courier, monospace;">run <- function() {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ssize <- 10000</span><br />
<span style="font-family: Courier New, Courier, monospace;"> nboot <- 1000</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x <- runif(ssize)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y <- replicate(nboot, sd(sample(x, ssize, TRUE)))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> sd(y)</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for (i in 1:4) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> cat(system.time(run())[3], " ")</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="background-color: #d9ead3; font-family: Courier New, Courier, monospace;"># result: 0.34 0.32 0.31 0.34</span><br />
<div>
<br /></div>
The direct translation to Julia gives:<br />
<span style="font-family: Courier New, Courier, monospace;">using Distributions</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">function run()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ssize = 10000</span><br />
<span style="font-family: Courier New, Courier, monospace;"> nboot = 1000</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x = rand(ssize)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y = Array(Float64,nboot)</span><br />
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:nboot</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y[i] = std(sample(x, ssize))</span><br />
<span style="font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"> std(y)</span><br />
<span style="font-family: Courier New, Courier, monospace;">end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for i in 1:4</span><br />
<span style="font-family: Courier New, Courier, monospace;"> print("$(@elapsed run()) ")</span><br />
<span style="font-family: Courier New, Courier, monospace;">end</span><br />
<span style="background-color: #d9ead3; font-family: 'Courier New', Courier, monospace;"># result: 0.38965987 0.331032088 0.3208918 0.315452803</span><br />
<div>
<br /></div>
<div>
So not counting longer Julia start up time - execution time is comparable.</div>
<div>
<br /></div>
<div>
Now we can try devectorizing Julia code:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">function run()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ssize = 10000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> nboot = 1000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> x = rand(ssize)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y = Array(Float64,nboot)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> bx = Array(Float64, ssize)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:nboot</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for j in 1:ssize</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> bx[j] = x[(rand(Uint32) % ssize) + 1]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y[i] = std(bx)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> std(y)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">for i in 1:4</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> print("$(@elapsed run()) ")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="background-color: #d9ead3; font-family: 'Courier New', Courier, monospace;"># result: 0.072204955 0.082288181 0.072243955 0.073718137</span></div>
</div>
<div>
<br /></div>
<div>
We have over 4-times speedup. So devectorization works perfectly and Julia lives up to the promise of performance gain.</div>
<div>
<br /></div>
<div>
However, this is not a whole story. Remember that GNU R performs extra work behinds the scenes that standard Julia ignores - it handles missing values (<span style="font-family: Courier New, Courier, monospace;">NA</span>) out of the box.</div>
<div>
<br /></div>
<div>
Actually we can tell Julia to take missing values into account via <span style="font-family: Courier New, Courier, monospace;">DataFrames</span> package. Let us test vectorized and devectorized code.</div>
<div>
<br /></div>
<div>
Vectorized Julia with NAs:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">using Distributions</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">using DataFrames</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">function run()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ssize = 10000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> nboot = 1000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> x = DataArray(rand(ssize))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y = DataArray(Float64,nboot)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:nboot</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y[i] = std(sample(x, ssize))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> std(y)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">for i in 1:4</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> print("$(@elapsed run()) ")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="background-color: #d9ead3; font-family: Courier New, Courier, monospace;"># result: 1.320962821 1.305368346 1.250524339 1.272740111</span></div>
</div>
<div>
<br /></div>
<div>
So w are over 4 times slower than GNU R now. How about devectorized Julia with NAs? Let us try:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">using DataFrames</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">function run()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ssize = 10000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> nboot = 1000</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> x = DataArray(rand(ssize))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y = DataArray(Array(Float64,nboot))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> bx = DataArray(Array(Float64, ssize))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for i in 1:nboot</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for j in 1:ssize</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> bx[j] = x[(rand(Uint32) % ssize) + 1]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> y[i] = std(bx)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> std(y)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">for i in 1:4</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> print("$(@elapsed run()) ")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end</span></div>
<div>
<span style="background-color: #d9ead3; font-family: Courier New, Courier, monospace;"># result: 1.027682685 0.994636538 1.017903246 1.021942776</span></div>
</div>
<div>
<br /></div>
<div>
There is a speed up, but it is minor and we still are 3 times slower.</div>
<div>
<br /></div>
<div>
Probably Julia code could be tuned, but we can draw some preliminary conclusions (at least for current versions of GNU R and Julia).</div>
<div>
<b>If you can use default Julia data types in your analysis Julia should easily beat GNU R in performance, especially after devectorization. However if you require handling of missing values in your code and have to use <span style="font-family: Courier New, Courier, monospace;">DataFrames</span> package in Julia you can expect GNU R to be quite well optimized.</b></div>
<div>
<br /></div>
<div>
PS.</div>
<div>
While trying to get a grasp of Julia I have collected some notes on its syntax and features from R-programmers perspective. You can have a peek its preliminary version <a href="http://bogumilkaminski.pl/files/julia_express.pdf">here</a>.</div>
Unknownnoreply@blogger.com10tag:blogger.com,1999:blog-4946490806848569840.post-60582346556813560792013-12-02T07:45:00.001-08:002013-12-02T07:45:24.997-08:00Speeding up model bootstrapping in GNU RAfter my <a href="http://rsnippets.blogspot.com/2013/11/simulatin-speed-gnu-r-vs-julia.html">last post</a> I have recurringly received two questions: (a) is it worthwhile to analyze GNU R speed in simulations and (b) how would simulation speed compare between GNU R and Python. In this post I want to address the former question and next time I am going to tackle the latter.<br />
<br />
<a name='more'></a>An area in which I use simulation in GNU R on a daily basis is bootstrapping. Therefore I have decided to check out how much speedup one can expect with tuning of standard model estimation procedures.<br />
<br />
I started with linear regression: a simple model with two explanatory variables and 100 observations. I wanted to compare speed of generation of 10000 bootstrap replications using standard <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">lm</span> function and bare matrix calculus formula for <b>b</b> solving the equation: X'X<b>b</b>=X'y. Here is the simulation:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">100</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x1 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x2 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x1 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x2 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rexp</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># add non-normal error distribution</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.m </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cbind</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"(Intercept)"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, x1, x2, y</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.f </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.frame</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x1, x2, y</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">boot.bare </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.m</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sample.int</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">replace</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> X </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">3</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tX </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">t</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">X</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">4</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">solve</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tX </span><span style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">%*%</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> X, tX </span><span style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">%*%</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> y</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">boot.lm </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">lm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">~</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">., </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">data</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.f</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sample.int</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">replace</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])$</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">coef</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rb </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">replicate</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, boot.bare</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">())[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># 0.73 seconds</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rl </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">replicate</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, boot.lm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># 16.00 seconds</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># check if both procedures produce the same results</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">range</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rb </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> rl</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;"># OK difference ranges from
-5.373479e-14 to 4.751755e-14</span><br />
<br />
Both formulas yield practically identical results but using <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">lm</span> produces over 20 times slower execution. Of course <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">lm</span> does much more work than simple calculation of parameter estimates - boot for bootstrapping I need only the parameters.<br />
<br />
So next I thought to check out a more complex model. Therefore I switched to logistic regression.<br />
I compared <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">glm</span> estimation with two alternatives: brute-force optimization of log-likelihood using <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">nlm</span> and direct computation of estimates using iteratively reweighted least squares (IRLS, the method actually used by <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">glm</span> internally). Here is the code:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">100</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x1 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x2 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x1 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x2 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1.5</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># add non-logistic
error distribution</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.m </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">cbind</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, x1, x2, y</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.f </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.frame</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x1, x2, y</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">boot.nlm </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> llik </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">a</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">exp</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">X </span><span lang="EN-US" style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">%*%</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> a</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">==</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">==</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">sum</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">log</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">prob</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.m</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sample.int</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> X </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">3</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tX </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">t</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">X</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">4</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">nlm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">llik, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">3</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">estimate<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">boot.irls </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.m</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sample.int</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> X </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">3</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tX </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">t</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">X</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> dm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">4</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> b </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">3</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">while</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">TRUE</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">exp</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">X </span><span lang="EN-US" style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">%*%</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> b</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pp </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> as.vector</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">prob </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pp </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rbind</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pp, pp, pp</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> grad </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tX </span><span lang="EN-US" style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">%*%</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prob</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> b </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> b </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">solve</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">((</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tX </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pp</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">%*%</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> X, grad</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">if</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">sum</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">abs</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">grad</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">6</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">return</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">b</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">boot.glm </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.f</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sample.int</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">glm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">., </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">data</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x, </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">family</span><span lang="EN-US" style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: grey; font-family: 'Courier New'; font-size: 10pt;">"binomial"</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">coef</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rn </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, boot.nlm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># 35.48 seconds</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># warnings related to procedure convergence produced</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ri </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, boot.irls</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">())[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># 5.49 seconds</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rg </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, boot.glm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># 34.68 seconds</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># check if all
procedures produce the same results</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">range</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rn </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> ri</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
-0.0002572192 0.0002470589</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">range</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rn </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> rg</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
-0.0002572191 0.0002470588</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">range</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">ri </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> rg</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">
</span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;"># -4.196647e-07 3.602730e-07</span><br />
<br />
We can see that using <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">nlm</span> is the slowest and produces unstable results and that IRLS and <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">glm</span> give identical results but IRLS is 6 times faster.<br />
<br />
What are the conclusions? If you really need it is is possible to substantially reduce computational time even for "core" functions.<br />
However, simplest means are not always guaranteed to be efficient (like <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">nlm</span> in logistic regression) and there is a question if the speedup pays of, as there is a substantial additional development effort needed.Unknownnoreply@blogger.com7tag:blogger.com,1999:blog-4946490806848569840.post-39470874512016086592013-11-23T15:24:00.000-08:002013-11-24T05:09:24.768-08:00Simulation speed: GNU R vs JuliaRecently there is a lot of noise about <a href="http://julialang.org/">Julia</a>. I have decided to test its speed in simulation tasks on my toy <a href="http://rsnippets.blogspot.com/2013/10/cont-model-part-ii.html">Cont model</a>. I thought I had vectorized my GNU R code pretty well, but Julia is much faster.<br />
<br />
<a name='more'></a>The model was described in my <a href="http://rsnippets.blogspot.com/2013/10/cont-model-part-ii.html">earlier posts</a> so let us go down to a comparison:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-rK_7iefgAqI/UpE3biAJHmI/AAAAAAAAAR0/8ax9KTk3nL0/s1600/rjulia.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="96" src="http://4.bp.blogspot.com/-rK_7iefgAqI/UpE3biAJHmI/AAAAAAAAAR0/8ax9KTk3nL0/s320/rjulia.png" width="320" /></a></div>
Here is my GNU R code:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">library</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e1071</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont.run </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in, reps, n,
d, l, s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sig </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rnorm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> mul </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">if</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sig </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> mul </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> mul </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">sum</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">></span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">abs</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">return</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">kurtosis</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> cont.run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">,
</span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">0.005</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">,
</span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">10.0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">0.01</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)))</span></b><br />
<br />
It's execution time is a bit below 10 seconds on my laptop.<br />
<br />
An equivalent Julia code is the following:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">using</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">Distributions</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont_run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn_in</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">Array</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">Float64</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">Array</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">Float64</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">aris</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">randn</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mul</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">if</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mul</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">if</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">></span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">aris</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ari</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">aris</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mul</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ari</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">j</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">if</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rand</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">j</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ari</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">kurtosis</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn_in</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">t_start</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">()</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">Array</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">Float64</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont_run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.005</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10.0</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">,</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.01</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">end</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><o:p></o:p></span></div>
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">println</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">()</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">-</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">
</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">t_start</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><br />
<br />
And on my machine it takes a bit less than 0.7 seconds to run.<br />
<br />
So we get over tenfold speedup. This is a significant difference for simulation experiments.<br />
I will have to dig more into Julia in the future.Unknownnoreply@blogger.com9tag:blogger.com,1999:blog-4946490806848569840.post-65784824165178948592013-11-14T07:53:00.001-08:002013-11-14T07:53:57.177-08:00Calibration of p-value under variable selection: an exampleVery often people report p-values for linear regression estimates after performing variable selection step. Here is a simple simulation that shows that such a procedure might lead to wrong calibration of such tests.<br />
<br />
<a name='more'></a>Consider a simple data generating process where y ~ N(0, 1) and x1, x2 ~ U(0,1). Assume that we take n=20 independent samples of each of these variables.<br />
We want to estimate model:<br />
<div style="text-align: center;">
y = a0 + a1 * x1 + a2 * x2 + error</div>
We know that true values of a1 and a2 are 0.<br />
<br />
I will use two procedures that are used to estimate the model:<br />
<ul>
<li>Full: we estimate the full model</li>
<li>Selection: we perform model selection using AIC criterion</li>
</ul>
<div>
We want to test whether parameters a1 and a2 are statistically significantly different than 0. </div>
<div>
In theory - if we set rejection threshold to <b>p</b> we expect that with probability <b>p</b> the null hypothesis will be rejected. I test this assertion by performing 10 000 Monte Carlo simulations of the model.<br />
<br />
Below is the procedure I have used. I assume here that if variable is removed from the model by AIC criterion then it is treated as insignificant in calculations of observed <b>p</b> (i.e. H0 is not rejected).<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">f </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">do.setep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">20</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x1 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x2 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rnorm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> m </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">lm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">~</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x1 </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">do.setep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> m </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">step</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">m, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">trace</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">summary</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">m</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)$</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">coef</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[-</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">4</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pvv </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">seq</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">by</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">prop </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">NULL</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">test </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">NULL</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">do.step </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">T, F</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> set.seed</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> p.x </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> do.call</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">replicate</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps, f</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">do.step</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, simplify </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> prop </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cbind</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">prop, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sapply</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pvv, </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">p.x </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">/</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> test </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cbind</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">test, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sapply</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pvv, </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
prop.test</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">p.x </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, pv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">p.value<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">par</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mfrow </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">matplot</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pvv, prop, type </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"l"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> xlab </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"p-value"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,
ylab </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"observed/expected"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">matplot</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pvv, test, type </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"l"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> xlab </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">"p-value"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, ylab </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">"prop.test
p-value"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><br />
<br />
and the plot it produces:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-rJpRPWStuuk/UoTuXRJwnFI/AAAAAAAAARQ/QC7dtwqP54I/s1600/n20.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://3.bp.blogspot.com/-rJpRPWStuuk/UoTuXRJwnFI/AAAAAAAAARQ/QC7dtwqP54I/s400/n20.png" width="400" /></a></div>
On the plots solid black line represents model with variable selection and dashed red line - model without selection. On both plots x-axis represents different values of <b>p</b> (threshold p-value ranging from 0.01 to 0.1; I use the same sample for different values of <b>p</b> to conserve simulation time - it would probably be more prudent to use separate samples to make observed values independent).<br />
<br />
The left plot shows us what is the relation of observed number of H0 rejections to theoretical number of rejections. It can be clearly seen that in model without variable selection the test of H0 is approximately properly calibrated (ratio is approximately equal to 1). However in model with variable selection there are over 10% more rejections than expected (too many times a variable is judged as significant).<br />
<br />
We can simply test if this bias is statistically significant. The results are shown on the right graph. The test shows that for model without variable selection the deviations are insignificant. On the other hand - for the model with variable selection we reject H0 that the observed number of rejections is unbiased.</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-15009964800966040152013-10-28T13:36:00.001-07:002014-06-27T16:53:55.826-07:00Cont model - Part IIIn my <a href="http://rsnippets.blogspot.com/2013/10/cont-model-back-after-year.html" target="_blank">last post</a> I have investigated properties of Cont model (you can download the paper <a href="http://www.proba.jussieu.fr/pageperso/ramacont/papers/clustering.pdf" target="_blank">here</a>). Today I would like to show how we can use simulations to further simplify its analysis.<br />
<br />
<a name='more'></a>First let us start with the observation that the model does not really require two parameters <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span> as they are directly linked. If we multiply <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> by 2 and divide <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span> by 2 we obtain exactly the same simulated returns path (scaled by 2). Let me motivate this result using simulation. The code that performs verification is simple (for explanation of the code logic look at my previous <a href="http://rsnippets.blogspot.com/2013/10/cont-model-back-after-year.html" target="_blank">post</a>):<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont.run </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in, reps, n,
d, l ,s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sig </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rnorm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">/</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">abs</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">return</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">100</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.002</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.01</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">5</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.01</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">m </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># comparison
multiplier</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">seeds </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># common random numbers seeds</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">range</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">mapply</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d, l, s, m, seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r1 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cont.run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d, l ,s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r2 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cont.run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1000</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> m, l </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> m ,s</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">range</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r1 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">/</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> m </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d, l, s, m, seeds</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)) </span></b><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># </span><span lang="EN-US" style="color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US;">-2.775558e-17 1.387779e-17</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<br />
We can see that <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">r1/m</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">r2</span> vectors are almost identical in all 100 simulations.<br />
<br />
This finding means that we can limit ourselves to analysis of <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d*l</span> product in simulation output analysis. So we go back to <a href="http://bogumilkaminski.pl/share/sim_output.txt" target="_blank">sim_output.txt</a> file and modify the visualization as follows:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">library</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">lattice</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> read.table</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"sim_output.txt"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">head</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> colClasses </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"numeric"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">4</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">dl </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cs </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">cut</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">seq</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.01</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cdl </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">cut</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">dl, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">seq</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.2</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">11</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sum.data </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">aggregate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cdl </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cs, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">data</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">mean</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">trellis.par.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">regions</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">list</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">col</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">topo.colors</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">100</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #0080ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">levelplot</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cdl</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cs, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">data</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sum.data,scales</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">list</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">list</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rot</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">90</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">xlab
</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">"d
* l"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, ylab </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">"s"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><br />
<br />
Here is the final plot showing average <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">k</span> as a function of <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d*l</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-ZCP_6EHPWac/Um7Dp1ZibmI/AAAAAAAAAQM/3cbVjUmaEMk/s1600/plot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-ZCP_6EHPWac/Um7Dp1ZibmI/AAAAAAAAAQM/3cbVjUmaEMk/s400/plot.png" height="326" width="400" /></a></div>
We can see that the model produces excess kurtosis when <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d*l</span> is small. What does it mean? In this case we have low <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> and low <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span>. So new incoming information has low standard deviation but daily returns are allowed to be large. The conclusion is that kurtosis will be present if there is low level of trading on the market (the sum <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">sum</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">]</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">></span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">)</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> <span style="color: navy;"><b>+</b></span></span><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">sum</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">]</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;"><</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">(-</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: navy; font-family: 'Courier New'; font-size: 10pt;">))</span></b> is relatively low).<br />
<br />
We will use simulation again to verify this hypothesis. Here is the code with added trading volume variable (<span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">t</span>):<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont.run.vol </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in, reps,
n, d, l ,s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sig </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rnorm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">t</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">/</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">t</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(-</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)))</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">/</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> n</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">abs</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">kurtosis</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">mean</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">t</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">burn.in</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]))</span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">100</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.001</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">5</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">20</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">mapply</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d, l, s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
cont.run.vol</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d, l ,s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d, l, s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">t</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">colnames</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"kurtosis"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"volume"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">par</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mar</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">4</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">4</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: #8000ff; font-family: 'Courier New'; font-size: 10pt;">plot</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b></div>
<br />
The resulting plot confirms our reasoning:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-koGlnAlpYTU/Um7IyaEaHOI/AAAAAAAAAQc/w6UYH-2mM8M/s1600/plot2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-koGlnAlpYTU/Um7IyaEaHOI/AAAAAAAAAQc/w6UYH-2mM8M/s320/plot2.png" height="319" width="320" /></a></div>
<br />
The final question, that I will leave open here, is whether the observed relationship is an artifact of the model or the relationship that currently I am working on to verify empirically.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-86047764610993345402013-10-16T15:28:00.001-07:002014-06-27T16:54:07.813-07:00Cont model back after a yearDuring <a href="http://essa2013.org/" target="_blank">ESSA2013</a> conference I had a discussion about Cont model I have commented <a href="http://rsnippets.blogspot.com/2012/11/essa2013-conference.html" target="_blank">a year ago</a>.<br />
In original <a href="http://www.proba.jussieu.fr/pageperso/ramacont/papers/clustering.pdf" target="_blank">paper</a> Cont highlights that his model produces distribution of returns characterized by positive excess kurtosis. In this post I want to investigate this assertion.<br />
<a name='more'></a>Cont simulation has three key parameters (check out the paper for details):<br />
<br />
<ul>
<li><span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span>: standard deviation of new public information incoming to the investors</li>
<li><span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span>: market depth (this implies that maximum percentage price change is in range [<span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">-1/l</span>,<span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">1/l</span>])</li>
<li><span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span>: percentage of investors updating their decision threshold</li>
</ul>
<br />
We want to simulate the model with 1000 agents for 11000 periods (dropping first 1000 as warm up) and calculate kurtosis of observed returns distribution. The selected ranges of <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span>, <span style="font-family: Courier New; font-size: x-small;">l</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span> parameters follow Cont recommendations. Here is the code that generates the data:<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">library</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e1071</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">library</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mgcv</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont.run </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in, reps,
n, d, l ,s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sig </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rnorm</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sum</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sig</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">/</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">*</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">abs</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">r</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">burn.in</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">reps</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">60000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.001</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">5</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">20</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">runif</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sim.points,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">mapply</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d, l, s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
kurtosis</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cont.run</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1000</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, d, l ,s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, d, l, s</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><br />
<br />
Warning! The simulation takes a lot of time so I have uploaded <a href="http://bogumilkaminski.pl/share/sim_output.txt" target="_blank">here the file sim_output.txt</a> containing the results. It contains four columns: <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span>, <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span>, <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span>, <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">k</span>, where last column holds calculated kurtosis.<br />
<br />
Now let us turn to the code that allows us to visually explore the data. I have saved the generated plots to GIF file using <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">animation</span> package. They show the probability of encountering excess kurtosis in generated data conditional on <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span>, <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span>.<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">library</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ggplot2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> read.table</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"sim_output.txt"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">head</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
colClasses </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">rep</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"numeric"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">4</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cd </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cut</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">d, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">seq</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.001</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cl </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cut</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">l, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">seq</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">5</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">20</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">16</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cs </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">cut</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">s, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">seq</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.01</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0.1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">p.excess </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> as.numeric</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">k </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sum.data </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">aggregate</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">p.excess </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">~</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cd </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cl </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cs, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">data</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">mean</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">levels</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sum.data</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cs</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)[</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">9</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">8</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">:</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">print</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">ggplot</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> geom_point</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">data</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sum.data</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">sum.data</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">$</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cs </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> i,</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> aes</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cl, y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> cd, colour </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> p.excess</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> shape </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">15</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, size </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">10</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
scale_colour_gradient</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">low </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"blue"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, high </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"red"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> theme</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">panel.background
</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
element_blank</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
axis.title.x </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> element_blank</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
axis.title.y </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> element_blank</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">()</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">text</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">element_text</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">colour </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"black"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, size </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">14</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
axis.text.x</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">element_text</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">angle </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">90</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">+</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> ggtitle</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">paste</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"cs:"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">}</span></b><br />
<br />
And here is the result (<span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> is on y-axis and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span> on x-axis):<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-9elgcWI19_E/Ukf9r634otI/AAAAAAAAAPE/ljFVOvP7r0c/s1600/animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-9elgcWI19_E/Ukf9r634otI/AAAAAAAAAPE/ljFVOvP7r0c/s400/animation.gif" height="240" width="400" /></a></div>
<br />
We can see that <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">s</span> is the least important parameter and there is a non linear interaction between <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span>. But most importantly - in the given range parameters excess kurtosis is not guaranteed to appear. In particular it does not happen when both <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">d</span> and <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">l</span> are large.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-79600159870316351232013-09-08T01:25:00.000-07:002013-09-27T00:19:50.026-07:00Visualizing optimization processOne of the approaches to graph drawing is application of so called <a href="http://en.wikipedia.org/wiki/Force-directed_graph_drawing">force-directed algorithms</a>. In its simplest form the idea is to layout the nodes on plane so that all edges in the graph have approximately equal length. This problem has very intuitive visualization so it is a nice case for showing how different optimization algorithms behave in high dimensions.<br />
<a name='more'></a>I want to position several balls (8 in the example below) on a plane in such a way that distance between all balls is approximately 0.5. Precisely my objective is to minimize the highest deviance from 0.5 of distance between any pair of balls.<br />
<br />
The interesting feature of this problem is that visualization of 8 balls on a plane is simple and intuitive - it is important because it allows me to visually assess the solution quality - not having to rely on objective function value only.<br />
On the other hand the optimization involves 16 decision variables for 8 balls and the objective function is not smooth and has many local minima - a perfect case to compare different optimization algorithms.<br />
<br />
The code that generates the plots for BFGS optimization algorithm is given below:<br />
<pre style="border: 1px solid rgb(204, 204, 204); font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; font-size: 12px; max-width: 95%; white-space: pre-wrap;"><code class="r" style="background-color: #f8f8f8; display: block; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; padding: 0.5em;"><span class="keyword" style="color: blue;">library</span><span class="paren" style="color: #687687;">(</span><span class="identifier">animation</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">b.init</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">balls</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">best.fit</span> <span class="operator" style="color: #687687;"><</span><span class="operator" style="color: #687687;"><-</span> <span class="operator" style="color: #687687;">+</span><span class="literal" style="color: #5848f6;">Inf</span>
<span class="identifier">pos</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">runif</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">2</span> <span class="operator" style="color: #687687;">*</span> <span class="identifier">balls</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">names</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">paste</span><span class="paren" style="color: #687687;">(</span><span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="string" style="color: #036a07;">"x"</span>, <span class="string" style="color: #036a07;">"y"</span><span class="paren" style="color: #687687;">)</span>, <span class="identifier">rep</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="identifier">balls</span>, <span class="identifier">each</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">2</span><span class="paren" style="color: #687687;">)</span>,
<span class="identifier">sep</span> <span class="operator" style="color: #687687;">=</span> <span class="string" style="color: #036a07;">""</span><span class="paren" style="color: #687687;">)</span>
<span class="keyword" style="color: blue;">return</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">b.plot</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">par</span><span class="paren" style="color: #687687;">(</span><span class="identifier">mar</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">2</span>, <span class="number" style="color: mediumblue;">2</span>, <span class="number" style="color: mediumblue;">1.5</span>, <span class="number" style="color: mediumblue;">0.5</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span class="comment" style="color: #4c886b;"># extract x and y coordinates of balls</span>
<span class="identifier">x</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">pos</span><span class="paren" style="color: #687687;">[</span><span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="literal" style="color: #5848f6;">T</span>, <span class="literal" style="color: #5848f6;">F</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">y</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">pos</span><span class="paren" style="color: #687687;">[</span><span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="literal" style="color: #5848f6;">F</span>, <span class="literal" style="color: #5848f6;">T</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">]</span>
<span class="comment" style="color: #4c886b;"># plot balls</span>
<span class="identifier">plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span>, <span class="identifier">y</span>, <span class="identifier">cex</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">5</span>, <span class="identifier">pch</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">20</span>, <span class="identifier">main</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">b.fit</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>,
<span class="identifier">xlim</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="identifier">min</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">0.1</span>, <span class="identifier">max</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">+</span> <span class="number" style="color: mediumblue;">0.1</span><span class="paren" style="color: #687687;">)</span>,
<span class="identifier">ylim</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="identifier">min</span><span class="paren" style="color: #687687;">(</span><span class="identifier">y</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">0.1</span>, <span class="identifier">max</span><span class="paren" style="color: #687687;">(</span><span class="identifier">y</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">+</span> <span class="number" style="color: mediumblue;">0.1</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span class="comment" style="color: #4c886b;"># and lines connecting them</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="paren" style="color: #687687;">(</span><span class="identifier">length</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">j</span> <span class="keyword" style="color: blue;">in</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="operator" style="color: #687687;">+</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">:</span><span class="identifier">length</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">lines</span><span class="paren" style="color: #687687;">(</span><span class="identifier">x</span><span class="paren" style="color: #687687;">[</span><span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="identifier">i</span>, <span class="identifier">j</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">]</span>, <span class="identifier">y</span><span class="paren" style="color: #687687;">[</span><span class="identifier">c</span><span class="paren" style="color: #687687;">(</span><span class="identifier">i</span>, <span class="identifier">j</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">]</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="paren" style="color: #687687;">}</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">b.fit</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="comment" style="color: #4c886b;"># our goal is to make distance between all balls equal to 0.5</span>
<span class="identifier">max</span><span class="paren" style="color: #687687;">(</span><span class="identifier">abs</span><span class="paren" style="color: #687687;">(</span><span class="identifier">dist</span><span class="paren" style="color: #687687;">(</span><span class="identifier">t</span><span class="paren" style="color: #687687;">(</span><span class="identifier">matrix</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span>, <span class="number" style="color: mediumblue;">2</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">0.5</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">f</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">fit</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">b.fit</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>
<span class="comment" style="color: #4c886b;"># update plot if improvement is found</span>
<span class="keyword" style="color: blue;">if</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">fit</span> <span class="operator" style="color: #687687;"><</span> <span class="identifier">best.fit</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">best.fit</span> <span class="operator" style="color: #687687;"><</span><span class="operator" style="color: #687687;"><-</span> <span class="identifier">fit</span>
<span class="identifier">b.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="keyword" style="color: blue;">return</span><span class="paren" style="color: #687687;">(</span><span class="identifier">fit</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">run</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">set.seed</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">pos</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">b.init</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">8</span><span class="paren" style="color: #687687;">)</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">20</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">b.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">pos</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">optim</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span>, <span class="identifier">f</span>, <span class="identifier">method</span> <span class="operator" style="color: #687687;">=</span> <span class="string" style="color: #036a07;">"BFGS"</span>, <span class="identifier">control</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">list</span><span class="paren" style="color: #687687;">(</span><span class="identifier">maxit</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">1000</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">$</span><span class="identifier">par</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">20</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">b.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">pos</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">ani.options</span><span class="paren" style="color: #687687;">(</span><span class="identifier">interval</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">0.05</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">saveGIF</span><span class="paren" style="color: #687687;">(</span><span class="identifier">run</span><span class="paren" style="color: #687687;">(</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
</code></pre>
It generates the following animation:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-auBiF8jNIGI/UiwwLIBq8zI/AAAAAAAAAN0/mjCSpoVfpAk/s1600/animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-auBiF8jNIGI/UiwwLIBq8zI/AAAAAAAAAN0/mjCSpoVfpAk/s400/animation.gif" width="400" /></a></div>
<br />
It is interesting to run this simulation for different initial positions of balls as BFGS might converge to different local minima or to try different optimization algorithms either from <span style="background-color: #f8f8f8; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; white-space: pre;">optim</span> function (like SANN) or any other optiization routine (function <span style="background-color: #f8f8f8; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; white-space: pre;">f</span> keeps track of best solution found independently from optimization procedure).Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4946490806848569840.post-23741765331849238892013-08-13T01:28:00.002-07:002013-08-13T01:28:57.266-07:00Genetic drift simulationWhile preparing for the new teaching semester I have created an implementation of NetLogo <a href="http://ccl.northwestern.edu/netlogo/models/GenDriftPlocal">GenDrift P local</a> in GNU R.<br />
<br />
<a name='more'></a>The model works as follows. Initially a square grid having <span style="font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; white-space: pre;">side</span> size is randomly populated with <span consolas="" console="" ejavu="" font-family:="" monaco="" mono="" monospace="" pre="" roid="" sans="" ucida="" white-space:="">n</span> types of agents (denoted by colors).<br />
Each agent randomly changes type (color) to one of its eight neighbors colors. In long run only one agent type remains due to random selection process (although it can take a long time if we set<br />
<span style="font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; white-space: pre;">side</span> to a large value).<br />
<br />
Here is the simulation code:<br />
<pre style="border: 1px solid rgb(204, 204, 204); font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; font-size: 12px; max-width: 95%; white-space: pre-wrap;"><code class="r" style="background-color: #f8f8f8; display: block; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; padding: 0.5em;"><span class="comment" style="color: #4c886b;"><span class="keyword" style="color: blue;">library</span><span class="paren" style="color: #687687;">(</span><span class="identifier" style="color: black;">lattice</span><span class="paren" style="color: #687687;">)</span><span style="color: black;">
</span><span class="keyword" style="color: blue;">library</span><span class="paren" style="color: #687687;">(</span><span class="identifier" style="color: black;">animation</span><span class="paren" style="color: #687687;">)</span></span></code><code class="r" style="background-color: #f8f8f8; display: block; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; padding: 0.5em;"><span class="comment" style="color: #4c886b;"># initialize simulation</span>
<span class="comment" style="color: #4c886b;"><span class="comment"># side - square grid size</span><span style="color: black;">
</span><span class="comment"><span class="comment"><span class="comment"># n - number of agent types (from </span>2 to 6)<span style="color: black;">
</span></span></span></span><span class="identifier">init</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">side</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">matrix</span><span class="paren" style="color: #687687;">(</span><span class="identifier">sample.int</span><span class="paren" style="color: #687687;">(</span><span class="identifier">n</span>, <span class="identifier">side</span> <span class="operator" style="color: #687687;">*</span> <span class="identifier">side</span>, <span class="literal" style="color: #5848f6;">TRUE</span><span class="paren" style="color: #687687;">)</span>, <span class="identifier">side</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="comment" style="color: #4c886b;"><span class="comment"># plot simulation state</span><span style="color: black;">
</span># s - simulation state</span>
<span class="comment" style="color: #4c886b;"># n - initial number of agent types</span>
<span class="identifier">do.plot</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">print</span><span class="paren" style="color: #687687;">(</span><span class="identifier">levelplot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">at</span> <span class="operator" style="color: #687687;">=</span> <span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">0</span><span class="operator" style="color: #687687;">:</span><span class="identifier">n</span><span class="paren" style="color: #687687;">)</span> <span class="operator" style="color: #687687;">+</span> <span class="number" style="color: mediumblue;">0.5</span>,
<span class="identifier">col.regions</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">1</span> <span class="operator" style="color: #687687;">+</span> <span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>,
<span class="identifier" style="background-color: transparent;">scales</span> <span class="operator" style="background-color: transparent; color: #687687;">=</span> <span class="identifier" style="background-color: transparent;">list</span><span class="paren" style="background-color: transparent; color: #687687;">(</span><span class="identifier" style="background-color: transparent;">draw</span> <span class="operator" style="background-color: transparent; color: #687687;">=</span> <span class="literal" style="background-color: transparent; color: #5848f6;">F</span><span class="paren" style="background-color: transparent; color: #687687;">)</span>, <span class="identifier">xlab</span> <span class="operator" style="color: #687687;">=</span> <span class="string" style="color: #036a07;">""</span>, <span class="identifier">ylab</span> <span class="operator" style="color: #687687;">=</span> <span class="string" style="color: #036a07;">""</span>,
<span class="identifier">colorkey</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">list</span><span class="paren" style="color: #687687;">(</span><span class="identifier">tick.number</span> <span class="operator" style="color: #687687;">=</span> <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span style="color: #687687;">}</span></code><code class="r" style="background-color: #f8f8f8; display: block; font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; padding: 0.5em;"><span class="comment" style="color: #4c886b;"><span class="comment"><span class="comment"># perform one step of simulation</span><span style="color: black;">
</span># state - simulation state</span></span>
<span class="identifier">sim.step</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">state</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">updated</span> <span class="operator" style="color: #687687;"><-</span> <span class="literal" style="color: #5848f6;">FALSE</span>
<span class="identifier">side</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">dim</span><span class="paren" style="color: #687687;">(</span><span class="identifier">state</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">[</span><span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">moves_x</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">rep</span><span class="paren" style="color: #687687;">(</span><span class="operator" style="color: #687687;">-</span><span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">1</span>, <span class="number" style="color: mediumblue;">3</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">[</span><span class="operator" style="color: #687687;">-</span><span class="number" style="color: mediumblue;">5</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">moves_y</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">rep</span><span class="paren" style="color: #687687;">(</span><span class="operator" style="color: #687687;">-</span><span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">1</span>, <span class="identifier">each</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">3</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">[</span><span class="operator" style="color: #687687;">-</span><span class="number" style="color: mediumblue;">5</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">sequence</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">sample.int</span><span class="paren" style="color: #687687;">(</span><span class="identifier">length</span><span class="paren" style="color: #687687;">(</span><span class="identifier">state</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="identifier">sequence</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">x</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">1</span> <span class="operator" style="color: #687687;">+</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">%%</span><span class="identifier">side</span>
<span class="identifier">y</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">1</span> <span class="operator" style="color: #687687;">+</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">%/%</span><span class="identifier">side</span>
<span class="identifier">ref</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">sample.int</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">8</span>, <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">ref.x</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">1</span> <span class="operator" style="color: #687687;">+</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">x</span> <span class="operator" style="color: #687687;">+</span> <span class="identifier">moves_x</span><span class="paren" style="color: #687687;">[</span><span class="identifier">ref</span><span class="paren" style="color: #687687;">]</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">%%</span><span class="identifier">side</span>
<span class="identifier">ref.y</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">1</span> <span class="operator" style="color: #687687;">+</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">y</span> <span class="operator" style="color: #687687;">+</span> <span class="identifier">moves_y</span><span class="paren" style="color: #687687;">[</span><span class="identifier">ref</span><span class="paren" style="color: #687687;">]</span> <span class="operator" style="color: #687687;">-</span> <span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span><span class="operator" style="color: #687687;">%%</span><span class="identifier">side</span>
<span class="keyword" style="color: blue;">if</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">state</span><span class="paren" style="color: #687687;">[</span><span class="identifier">x</span>, <span class="identifier">y</span><span class="paren" style="color: #687687;">]</span> <span class="operator" style="color: #687687;">!=</span> <span class="identifier">state</span><span class="paren" style="color: #687687;">[</span><span class="identifier">ref.x</span>, <span class="identifier">ref.y</span><span class="paren" style="color: #687687;">]</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">state</span><span class="paren" style="color: #687687;">[</span><span class="identifier">x</span>, <span class="identifier">y</span><span class="paren" style="color: #687687;">]</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">state</span><span class="paren" style="color: #687687;">[</span><span class="identifier">ref.x</span>, <span class="identifier">ref.y</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">updated</span> <span class="operator" style="color: #687687;"><-</span> <span class="literal" style="color: #5848f6;">TRUE</span>
<span class="paren" style="color: #687687;">}</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">list</span><span class="paren" style="color: #687687;">(</span><span class="identifier">state</span>, <span class="identifier">updated</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">go</span> <span class="operator" style="color: #687687;"><-</span> <span class="keyword" style="color: blue;">function</span><span class="paren" style="color: #687687;">(</span><span class="identifier">size</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">s</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">init</span><span class="paren" style="color: #687687;">(</span><span class="identifier">size</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">updated</span> <span class="operator" style="color: #687687;"><-</span> <span class="literal" style="color: #5848f6;">TRUE</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">4</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">do.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">i</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">0</span>
<span class="keyword" style="color: blue;">while</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">updated</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">i</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">i</span> <span class="operator" style="color: #687687;">+</span> <span class="number" style="color: mediumblue;">1</span>
<span class="identifier">do.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">out</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">sim.step</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">s</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">out</span><span class="paren" style="color: #687687;">[</span><span class="paren" style="color: #687687;">[</span><span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">]</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">updated</span> <span class="operator" style="color: #687687;"><-</span> <span class="identifier">out</span><span class="paren" style="color: #687687;">[</span><span class="paren" style="color: #687687;">[</span><span class="number" style="color: mediumblue;">2</span><span class="paren" style="color: #687687;">]</span><span class="paren" style="color: #687687;">]</span>
<span class="identifier">do.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">print</span><span class="paren" style="color: #687687;">(</span><span class="identifier">i</span><span class="paren" style="color: #687687;">)</span>
<span class="keyword" style="color: blue;">for</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">i</span> <span class="keyword" style="color: blue;">in</span> <span class="number" style="color: mediumblue;">1</span><span class="operator" style="color: #687687;">:</span><span class="number" style="color: mediumblue;">4</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">do.plot</span><span class="paren" style="color: #687687;">(</span><span class="identifier">s</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span>
<span class="paren" style="color: #687687;">}</span>
<span class="identifier">set.seed</span><span class="paren" style="color: #687687;">(</span><span class="number" style="color: mediumblue;">1</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">animate</span> <span class="operator" style="color: #687687;"><-</span> <span class="literal" style="color: #5848f6;">FALSE</span> <span class="comment" style="color: #4c886b;"># do we want to save animation?</span>
<span class="identifier">side</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">15</span> <span class="comment" style="color: #4c886b;"># simulation grid will be side x side</span>
<span class="identifier">n</span> <span class="operator" style="color: #687687;"><-</span> <span class="number" style="color: mediumblue;">5</span> <span class="comment" style="color: #4c886b;"># initial number of agent types</span>
<span class="keyword" style="color: blue;">if</span> <span class="paren" style="color: #687687;">(</span><span class="identifier">animate</span><span class="paren" style="color: #687687;">)</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">ani.options</span><span class="paren" style="color: #687687;">(</span><span class="identifier">interval</span> <span class="operator" style="color: #687687;">=</span> <span class="number" style="color: mediumblue;">0.2</span><span class="paren" style="color: #687687;">)</span>
<span class="identifier">saveGIF</span><span class="paren" style="color: #687687;">(</span><span class="identifier">go</span><span class="paren" style="color: #687687;">(</span><span class="identifier">side</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span> <span class="keyword" style="color: blue;">else</span> <span class="paren" style="color: #687687;">{</span>
<span class="identifier">go</span><span class="paren" style="color: #687687;">(</span><span class="identifier">side</span>, <span class="identifier">n</span><span class="paren" style="color: #687687;">)</span>
<span class="paren" style="color: #687687;">}</span></code></pre>
We can select if we want to see the animation on the screen or save it to file via <span style="font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; white-space: pre;">animate</span> option.<br />
The animation the code generates is presented below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-z2NG0yUCJWc/UgnrXBxy2NI/AAAAAAAAAM4/Aivw2vkHy-k/s1600/animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-z2NG0yUCJWc/UgnrXBxy2NI/AAAAAAAAAM4/Aivw2vkHy-k/s400/animation.gif" width="400" /></a></div>
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4946490806848569840.post-44917850266706373252013-06-28T05:44:00.000-07:002013-06-28T12:19:46.778-07:00Testing function arguments in GNU RRecently I have read a nice <a href="http://www.r-bloggers.com/practicing-static-typing-in-r-prime-directive-on-trusting-our-functions-with-object-oriented-programming/">post</a> on ensuring that proper arguments are passed to a function using GNU R class system. However, I often need a more lightweight solution to repetitive function argument testing.<br />
<a name='more'></a>The alternative idea is to test function arguments against a specified pattern given in a string. The pattern I use has the form:<br />
<span style="font-family: Courier New, Courier, monospace;">([argument type][required argument length])+</span>.<br />
The allowed argument types are:<br />
<span style="font-family: Courier New, Courier, monospace;"> b logical</span><br />
<span style="font-family: Courier New, Courier, monospace;"> i integer</span><br />
<span style="font-family: Courier New, Courier, monospace;"> d double</span><br />
<span style="font-family: Courier New, Courier, monospace;"> n numeric</span><br />
<span style="font-family: Courier New, Courier, monospace;"> c complex</span><br />
<span style="font-family: Courier New, Courier, monospace;"> s character</span><br />
<span style="font-family: Courier New, Courier, monospace;"> f function</span><br />
<span style="font-family: Courier New, Courier, monospace;"> l list</span><br />
<span style="font-family: Courier New, Courier, monospace;"> a any type</span><br />
<div>
and argument lengths are:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ? 0 or 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> * 0 or more</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> + 1 or more</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> n exactly n</span></div>
</div>
<div>
<br /></div>
<div>
So for example pattern <span style="font-family: Courier New, Courier, monospace;">"n2f?s+"</span> means that function requires three arguments: one numeric vector of length 2, a function (or nothing) and character vector of any positive length.</div>
<div>
<br /></div>
<div>
The function that performs checking of the arguments against the pattern is given here:</div>
<div>
<br /></div>
<div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule, ...,
empty.null </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">TRUE</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> err </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">function</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">type, position, required, found</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> e </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">FALSE</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">attr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"type"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> type<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">attr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"position"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> position<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">attr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"required"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> required<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">attr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"found"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> found<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">e</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stopifnot</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">require</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">stringr, quietly </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> arglist </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">list</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">...</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(!(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.character</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">||</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">||</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">nchar</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: rule must be a nonempty string"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> type </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> str_locate_all</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"[bidncsfla]"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)[[</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> count </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> invert_match</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> type </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> str_sub</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule, type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> count </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> str_sub</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rule, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,</span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">!=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">""</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: rule must start with type
[bidncsfl]"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> count </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"><-</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[-</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">!=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: number of type and count specifiers
must be equal"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> seq_along</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">((!</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">%in%</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"?"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"*"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"+"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">&&</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">regexpr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"^[1-9][0-9]*$"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">paste</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: unrecognized count"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">!=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: number of type specifiers must be
equal to number of variables"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">for</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">in</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> seq_along</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">((!</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">%in%</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"?"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"*"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"+"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">&&</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">regexpr</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"^[1-9][0-9]*$"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">-</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">stop</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">paste</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"improper rule: unrecognized count"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">((</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #804000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">%in%</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">c</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"?"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"*"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">&&</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.null</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">&&</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
empty.null</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">next</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(!</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">switch</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> b </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.logical</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> i </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.integer</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> d </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.double</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> n </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.numeric</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">c</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.complex</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> s </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.character</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> f </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.function</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> l </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> is.list</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> a </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">TRUE</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">err</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"type"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,
i, type</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">typeof</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"?"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">></span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">err</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"count"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,
i, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">else</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"*"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># nothing to do - always met</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">else</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"+"</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">==</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">0</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">err</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"count"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,
i, count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">else</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">if</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">!=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> as.integer</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">{</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">err</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"count"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,
i, as.integer</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">count</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">])</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">length</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">arglist</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">[[</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">]])))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">return</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">TRUE</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">}</span></b></div>
<div>
<br /></div>
<div>
Its first argument is <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">rule</span> a pattern that arguments should meet. Next the arguments are passed. For example in the following calls:</div>
<div>
<br /></div>
<div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"b1i1d1n1c1s1f1l1a1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">L, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1.0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sin</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">list</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">raw</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># TRUE</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"b1i1d1n1c1s1f1l1a1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">L, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1.0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">sin</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">list</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">raw</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;"># FALSE</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"b1i1d1n1c1s1f1l1a1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> T, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">L, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1.0</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">i, </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">"1"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">sin</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">, </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">raw</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">(</span></b><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">1</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">))</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">#
FALSE</span></div>
<div>
<br /></div>
<div>
the first is returns <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">TRUE</span> and second and third return <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">FALSE</span> (in the first there an improper length of last argument and in the second improper variable type). In case when <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">FALSE</span> is returned it contains attributes with diagnostic information on validation error type.</div>
<div>
<br /></div>
<div>
Additionally the function has an optional parameter <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">empty.null</span>. It influences the way it handles variables of length 0 (which are allowed when <span style="font-family: Courier New, Courier, monospace;">"?"</span> or <span style="font-family: Courier New, Courier, monospace;">"*"</span> argument length constraint is used). If it is set to <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">TRUE</span> then <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">NULL</span> as an argument is accepted as valid. You can see this in action in the following code:</div>
<div>
<br /></div>
<div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"l?"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">NULL</span></b><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: green; font-family: 'Courier New'; font-size: 10pt;"># TRUE</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">is.valid</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">(</span></b><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"l?"</span><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">, </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">NULL</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt;">, </span><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;">empty.null </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">=</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><b><span style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">FALSE</span><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: PL; mso-fareast-theme-font: minor-fareast; mso-highlight: white;">)</span></b><span style="background-color: white; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"> </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; color: green; font-family: 'Courier New'; font-size: 10pt; line-height: 115%;"># FALSE</span></div>
</div>
<div>
<br /></div>
<div>
In summary <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">is.valid</span> function is a simple and compact way to check type and length of passed arguments and can be considered as an alternative to the approach proposed in the <a href="http://www.r-bloggers.com/practicing-static-typing-in-r-prime-directive-on-trusting-our-functions-with-object-oriented-programming/">post</a> I have mentioned above.</div>
<div>
<br /></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4946490806848569840.post-40724505414791785032013-05-31T07:36:00.001-07:002013-05-31T07:36:41.469-07:00Regression regularization exampleRecently I needed a simple example showing when application of regularization in regression is worthwhile. Here is the code I came up with (along with basic application of parallelization of code execution).
<br />
<br />
<a name='more'></a>Assume you have 60 observations and 50 explanatory variables <i>x1</i> to <i>x50</i>. All these variables are IID from uniform distribution on interval [0, 1). Predicted variable <i>y</i> is generated as a sum of variables <i>x1</i> to <i>x50</i> and independent random noise <i>N</i>(0, 1).<br />
Our objective is to compare for such data: (a) linear regression on all 50 variables, regressions obtained by variable selection using (b) AIC and (c) BIC criteria and (d) Lasso regularization.<br />
What we do is generate 100 times the training data set and compare the four predictions against known expected value of <i>y</i> for 10 000 randomly selected values of explanatory variables. We use mean squared deviation of the prediction from the mean (thus for ideal model it is equal to 0).<br />
<br />
Here is the code that runs the simulation. Because each step of the procedure is lengthily I parallelize the computations.<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">library</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #0080ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">parallel</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">run </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">job</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">require</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">lasso2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
gen.data </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">v, n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.frame</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">v, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># true y is equal to sum of x</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> rowSums</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">names</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">paste</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"x"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">v, sep </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">""</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"y"</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">return</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
v </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">50</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
n </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">60</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
data.set </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> gen.data</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">v, n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># add
noise to y in training set</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rnorm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
new.set </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> gen.data</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">v, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
model.lm </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">lm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> ., data.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
model.aic </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">step</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">model.lm, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">trace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
model.bic </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">step</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">model.lm, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">trace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, k </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">log</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
model.lasso </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> l1ce</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> ., data.set,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> sweep.out </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">NULL</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, standardize </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">FALSE</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
models </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">list</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">model.lm,
model.aic, model.bic, model.lasso</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
results </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">numeric</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">length</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">models</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">j </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> seq_along</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">models</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pred </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">predict</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">models</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">j</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, newdata </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> new.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> results</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">j</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">]</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">mean</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">((</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pred </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> new.set</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">^</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">return</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">results</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cl </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> makeCluster</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">4</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">system.time</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">msd </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">t</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">parSapply</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cl, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">100</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># 58.07 seconds</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">stopCluster</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">cl</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">colnames</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">msd</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"lm"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"aic"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"bic"</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"lasso"</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">par</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">mar </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">boxplot</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">msd</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">:</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">ncol</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">msd</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">lines</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">i </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.4</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, i </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0.4</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rep</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">mean</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">msd</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">[</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, i</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">])</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">col</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">"red"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, lwd </span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">=</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">2</span><b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">)</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-highlight: white;">}</span></b><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<br />
The code produces boxplots for distribution of mean squared deviation from theoretical mean and additionally puts red line at the mean level of mean squared deviation. Here is the result:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-eliZ-hKFMCY/Uaiz7hQ4HEI/AAAAAAAAAJ8/Z9YizchrEHA/s1600/compare.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="319" src="http://1.bp.blogspot.com/-eliZ-hKFMCY/Uaiz7hQ4HEI/AAAAAAAAAJ8/Z9YizchrEHA/s320/compare.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: start;">
Notice that in this example neither AIC not BIC improve over linear regression with all variables. However Lasso consistently produces significantly better models.</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4946490806848569840.post-4689789385888568232013-05-20T08:14:00.002-07:002013-05-20T08:14:37.464-07:00Model fitting exam problemRecently I have run an exam where the following question had risen many problems for students (here I give its shortened formulation). You are given the data generating process <i>y</i> = 10<i>x</i> + <i>e</i>, where <i>e</i> is error term. Fit linear regression using <span style="background-color: white; color: #8000ff; font-family: 'Courier New'; font-size: 13px;">lm</span>, neural net using <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">nnet</span> with size equal to 2 and 10 and regression tree using <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">rpart</span>. What can be said about distribution of prediction error of such four modeling techniques?<br />
<a name='more'></a><br/>Here is the code that generates the required comparison assuming that <i>x</i> ~ U(0, 1) and <i>e</i> ~ N(0, 1) and two example levels of training sample size 20 and 200.<br />
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">library</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">rpart</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">library</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">nnet</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">run </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
x </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">runif</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">+</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">rnorm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
new.x </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> data.frame</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">seq</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">0</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, len </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10000</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
models </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">list</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">linear </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">lm</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> tree </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> rpart</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> nnet2 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> nnet</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x, size </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">trace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F, linout </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> nnet10 </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> nnet</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">y </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">~</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> x, size </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">trace</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> F, linout </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> T</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">sapply</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">models, </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">function</span></b><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">model</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> pred </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"><-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">predict</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">model, newdata </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">=</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> new.x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">sum</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">((</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">pred </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">-</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">10</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">*</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> new.x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">$</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">x</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">^</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">2</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">})</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">set.seed</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">1</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">for</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n </span><b><span lang="EN-US" style="background: white; color: blue; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">in</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">c</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">20</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, </span><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">200</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"> </span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">{</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">cat</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"--- n
="</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, n, </span><span lang="EN-US" style="background: white; color: grey; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">"---\n"</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">
</span><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">print</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">summary</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">t</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #8000ff; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">replicate</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background: white; color: #ff8000; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">100</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">, run</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">(</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;">n</span><b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">)))))</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-US" style="background: white; color: navy; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">}</span></b><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># --- n = 20 ---</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># linear tree nnet2 nnet10 </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Min. : 17.32
Min. :21046 Min.
: 322.9 Min.
: 566</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
1st Qu.: 247.25 1st
Qu.:22562 1st Qu.: 1753.1 1st Qu.:
5759</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Median : 725.22 Median
:24537 Median : 3419.2 Median :
10961</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Mean :1071.07 Mean
:25644 Mean : 7221.4
Mean : 87200</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
3rd Qu.:1651.43 3rd
Qu.:27559 3rd Qu.: 6877.1 3rd Qu.:
22494</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Max. :6614.57 Max.
:40742 Max. :84169.8
Max. :4309641</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># --- n = 200 ---</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;"># linear tree nnet2 nnet10 </span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Min. : 1.107
Min. :1976 Min.
: 32.62 Min.
: 119.7</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
1st Qu.: 25.939 1st
Qu.:2851 1st Qu.: 183.82
1st Qu.: 313.4</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Median : 76.533 Median
:3366 Median : 293.65
Median : 531.5</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
Mean :112.766 Mean
:3490 Mean : 2008.36
Mean : 2211.1</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-US" style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; mso-ansi-language: EN-US; mso-highlight: white;">#
3rd Qu.:160.217 3rd
Qu.:3921 3rd Qu.: 479.10
3rd Qu.: 742.3</span><span lang="EN-US" style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: 'Courier New'; font-size: 10pt;"><o:p></o:p></span></div>
<span style="background: white; color: green; font-family: "Courier New"; font-size: 10.0pt; line-height: 115%; mso-ansi-language: PL; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">#
Max. :568.374 Max.
:6502 Max. :83603.10
Max. :83444.6</span><br />
<br />
It is simple that linear regression is optimal as it is properly specified. Next in general neural net with size 2, neural net with size 10 and regression tree follow. The reason is that neural nets use S-shaped transformations and have effectively more parameters than are needed to fit the relationship. Finally regression tree is simply not well suited for modeling linear relationships between variables.<br />
<br />
However, neural nets are initialized using random parameters and sometimes BFGS optimization fails and very poor fits can occur. This can be seen by large values of <span style="background-color: white; color: green; font-family: 'Courier New'; font-size: 13px; line-height: 14px;">Max.</span> in <span style="background-color: white; color: green; font-family: 'Courier New'; font-size: 13px;">nnet2</span> and <span style="background-color: white; color: green; font-family: 'Courier New'; font-size: 13px;">nnet10</span>. The median of the results is largely unaffected by this but evaluation of mean expected error is very unstable due to the outliers (in order to get more reliable estimates more than 100 replications are needed).<br />
<br />
Of course by modifying <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">rpart</span> or <span style="background-color: white; font-family: 'Courier New'; font-size: 13px;">nnet</span> one can get a bit different results but the general conclusions will be similar.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4946490806848569840.post-58895743653346433142013-04-30T13:42:00.000-07:002013-04-30T13:42:03.703-07:00Missing tikzDeviceI love using <span style="font-family: Courier New, Courier, monospace;">tikzDevice</span>. When preparing LaTeX documents I switched to prepare all graphs in GNU R and then port them to TeX using <span style="font-family: Courier New, Courier, monospace;">tikzDevice</span>. Recently I have moved to GNU R 3.0.0 and was shocked to find that this package is no longer available on CRAN.<br />
<a name='more'></a>Fortunately I was able to find package on the web. Unfortunately - installing old package on GNU R version 3.0.0 simply generates an error when trying to load it.<br />
<br />
So for now the only solution for me was to:<br />
<br />
<ul>
<li>Install GNU R 2.15.3 in parallel to new one (fortunately this is not a problem)</li>
<li>downloading <span style="font-family: Courier New, Courier, monospace;">tikzDevice</span> 0.6.3 which can be found at <a href="http://download.r-forge.r-project.org/bin/windows/contrib/2.15/">http://download.r-forge.r-project.org/bin/windows/contrib/2.15/</a></li>
<li>installing <span style="font-family: 'Courier New', Courier, monospace;">tikzDevice</span> manually from local disk</li>
<li>installing <span style="font-family: Courier New, Courier, monospace;">filehash</span> package that it requires</li>
</ul>
Fortunately this worked.<br />
<br />
I really hope that <span style="font-family: 'Courier New', Courier, monospace;">tikzDevice</span> will be back to CRAN in the future.Unknownnoreply@blogger.com5