# Migration of DistWin analysis structure to R

The Windows-based GUI software for conducting distance sampling analysis (*Distance for Windows*, heretofore *DistWin*) has been the industry-standard for two decades.  Our development work continues to incorporate state-of-the-art analytical methods for software users.  Most of that software development takes place in the `R` language.

The `readdst` package was created for statisticans and biologists who have existing `DistWin` projects and wish to port them into the `R` environment.  It was originally intended as an *in-house* testing tool to enable comparison of analysis results of newly-developed analysis software, with results produced by `DistWin`. The purpose of the package is to convert data and analyses stored in `DistWin` projects into data objects and function calls for analysis using `R`.

This vignette is intended to give a few examples of the capabilities of `readdst`.  Readers of this vignette should have some familiarity with the organisation of `DistWin` projects along with functions in the `mrds` library for fitting detection functions.

Note too, `readdst` can only work with `DistWin` projects created by `DistWin` versions 6 or 7.

# Structure of `DistWin` project

`DistWin` projects carry not only the data collected in the field, but also survey descriptions, data filters, model definitions and analysis results.  All of these facets are stored in Access database files created by `DistWin`.  Reading the Access files into `R` is at the heart of `readdst`.

* Survey information includes point or line transect types, whether animals were recorded as individuals or clusters, whether perpendicular distances or radial distances and angles were recorded.
* Data filters possess information about sightings to include in analysis (truncation distance, species indicators for multi-species surveys)
* Model definitions describe the key function and adjustment term combinations to be fitted as the detection function, whether data will be combined across strata for detection function fitting, etc.
* Analysis results are the product of submitting the survey/filter/model combination to the analysis engines in `DistWin`.

# Example project files to transform

There is a project that comes with the distribution of `DistWin` located in `My Documents\My Distance Projects\Sample Projects` We will use to demonstrate the use of `DistWin`.  In addition, we have taken another project that ships with `DistWin`, `Stratify example`, renamed it `Vignette-stratify` and added further analyses to that project for use in this vignette.  This third example ships with the `readdst` R package and is found in the library folder associated with the `readdst` package (see below).

> Purists among the readers will recognise that spaces in directory names can cause problems for `R`.  If you will be converting `DistWin` projects located in other locations on a Windows machine, or are working under a different operating system where spaces in pathnames are forbidden, then you may need to make some alterations to the example code provided in this vignette.

```{asis, echo=FALSE}
## Ducknest
This project consists of an artificial example in which duck nest objects are collected as *binned* data.  The units of the project are non-SI miles and feet.  There is one non-default data filter, specifying a 6 foot perpendicular truncation distance.  There are four model definitions present:

* half-normal with cosine adjustment
* hazard rate with polynomial adjustment
* uniform with cosine adjustment, and
* half-normal with Hermite adjustment.
```

## Vignette-stratify

Synthetic line transect data, units of measure in non-SI units (effort and perpendicular distance in nautical miles).  Animals were detected in groups, so cluster size enters into the calculation of animal density.  The project contains a multiplier derived from an external estimate of $\widehat{g(0)}$, but for purposes of this vignette, the use of the multiplier has been removed from the `DistWin` analysis.

The study area also contains two types of habitat and the design incorporated habitat information.  Consequently, the brief analysis focuses upon whether it is most parsimonious to fit habitat-specific detection functions, or whether a single detection function pooled across habitat types.

## Amakihi

This project demonstrates the use of the MCDS (multiple covariate) analysis engine in conjunction with a point transect sampling survey (using SI units).  The analyses present in the project represent 16 model results presented in Marques et al. (2007).  All analyses employ a data filter specifying an 82.5m truncation distance.

# Conversion process

The following code chunk performs some preliminary setup before conducting any conversions.  First, if running on a Windows machine, the code checks that the 32-bit version of R is being used.  R uses the `RODBC` package to read Access files; this package is only supported by 32-bit versions of R.  On a Mac OS X or Linux/Unix machine, `readdst` makes use of [`mdb-tools`](http://mdbtools.sourceforge.net) and the `Hmisc` `R` package.

Second, a path to the `Sample Projects` directory is created for subsequent use when accessing projects that ship with `DistWin`.

```{r, setup}
library(readdst)
if (.Platform$OS.type=="windows" && R.Version()$platform!="i386-w64-mingw32") print("32-bit version of R needed to run RODBC")
home.dir <- path.expand("~")
sample.proj.dir <- paste0(home.dir, "/My Distance Projects/Sample Projects/")
```

```{r, convert-ducknest, echo=FALSE, eval=FALSE}
ducknest.proj.file <- paste0(sample.proj.dir, "Ducknest")
ducknest.proj <- convert_project(ducknest.proj.file)
```

The Amakihi project that ships with `DistWin` and resides in `My Documents\My Distance Projects\Sample Projects` is converted with these two commands.  There are no arguments to the function `convert_project()` other than the absolute path to the `DistWin` project (without the `.dst` file extension).
```{r, convert-amakihi}
amakihi.proj.file <- paste0(sample.proj.dir, "Amakihi")
amakihi.proj <- convert_project(amakihi.proj.file)
```


The `Vignette-stratify` project that ships with `readdst` and resides wherever `R` libraries are stored, is covered with these three commands.
```{r, convert-stratify}
stratify.proj.directory <- system.file("Stratify", package="readdst")
stratify.proj.name <- paste0(stratify.proj.directory, "/Vignette-stratify")
stratify.proj <- convert_project(stratify.proj.name)
```

## Components of converted `DistWin` projects

### Analyses

The object created by `convert_project()` is a named list.  There are as many elements in the list as analyses (run or not run) defined in the `DistWin` project.  The names of the list elements correspond to the names of the analyses:

```{r, stratify-list-names, comment=NA}
length(stratify.proj)
names(stratify.proj)
```

Each analysis is itself a named list, with list elements describing such things as data filter used and the equivalent call to  `ddf()` from the `mrds` library to fit the detection function specified in the analysis.  Several of these components will be discussed later.

### Data for each analysis

Stored with each analysis are the data that were used in the analysis.  Recall that `DistWin` allowed seperate specification of model definitions and data filters, which could be coupled to form an analysis.  The data (in two forms) as well as the measurement units for effort, perpendicular (radial) distances and area are all stored in the `env` element of each analysis list element[^environment].

```{r, stratify-environment, comment=NA}
ls(stratify.proj$'Half-normal cosine no stratification exact'$env)
```

Data are stored both in the `flatfile` format with all levels of the data hierarchy in a single object,  
```{r, stratify-data, echo=FALSE}
knitr:::kable(head(stratify.proj$'Half-normal cosine no stratification exact'$env$data, n = 3))
```

or in the multiple object form with a separate data frame for the `region.table`, `sample.table` and `obs.table`.


## Analysis of converted project

The `convert_project()` function makes the data for a distance sampling analysis available and transforms the syntax of detection function fitting into calls to `ddf()`.  This makes it possible to:

* repeat the analyses performed by `DistWin` in `R`, and/or
* conduct new analyses of data that have been stored in `DistWin`.

### Repeating an analysis

Recall each analysis from a `DistWin` project is represented as a list element of an object created by `convert_project()`.  The function `run_analysis()` takes as an argument an analysis and conducts that analysis in `R` using `ddf()` to fit a detection function and produce estimates of detection probability and $\hat{N}$ in the covered region.

```{r, stratify-ddf, comment=NA}
library(mrds)  # to access GOF function and dht()
stratify.reanalyse <- run_analysis(stratify.proj$'Half-normal cosine no stratification exact')
plot(stratify.reanalyse, showpoints=FALSE, pl.den=0)
gof.test <- ddf.gof(stratify.reanalyse, qq=FALSE)
text(1.2,0.9, cex=0.6,
     paste("K-S GOF D=",round(gof.test$dsgof$ks$Dn, 4), "\nP=", round(gof.test$dsgof$ks$p,3)))
summary(stratify.reanalyse)
```

Producing estimates of abundance from the fitted `ddf()` model requires a call to `dht()`.

```{r, stratify-dht}
stratify.abund <- dht(stratify.reanalyse, 
                      region = stratify.proj$`Half-normal cosine no stratification exact`$env$region.table, 
                      sample = stratify.proj$`Half-normal cosine no stratification exact`$env$sample.table, 
                      obs = stratify.proj$`Half-normal cosine no stratification exact`$env$obs.table)
```

```{r, dht-tables, echo=FALSE}
knitr::kable(stratify.abund$individuals$summary)
knitr::kable(stratify.abund$individuals$N[1:6])
```

The call to `dht()` shows the use of the hierarchial data type (region, sample and observation tables) available within each analysis.

It would be possible to perform new analyses of data present in `DistWin` projects, such as fitting a hazard-rate key function to the `stratify` data set that had only been fitted with half-normal detection functions.

```{r, stratify-hazard}
stratify.hazard <- ddf(dsmodel=~cds(key="hr", formula=~1, adj.series="cos", adj.order=2),
                   meta.data=list(width=1.94000005722046,left=0), 
                   control=list(mono=TRUE, mono.strict=TRUE), method="ds", 
                   data=stratify.proj$'Half-normal cosine no stratification exact'$env$data)
hazard.aic <- stratify.hazard$criterion
halfnorm.aic <- stratify.reanalyse$criterion
```

This hazard rate detection function can also be fitted by an equivalent call to the `ds()` function found in the `Distance` package available on CRAN.  `ds()` serves as a wrapper for `ddf()`.  We show both the `ddf()` (created by `run_analysis()` in `readdst`) and `ds()` so readers can select the approach they desire.

```{r, ds-strat-haz, eval=FALSE}
stratify.hazard <- ds(key="hr", formula=~1, adj.series="cos", adj.order=2,
                   truncation=1.94000005722046,
                   data=stratify.proj$'Half-normal cosine no stratification exact'$env$data)
```

The hazard rate model (with a cosine adjustment) had an AIC of `r round(hazard.aic,1)` while the unadjusted half-normal model had an AIC of `r round(halfnorm.aic,1)` suggesting we were on firm footing when fitting the half-normal model to the `stratify` dataset.

# Comparing results of `DistWin` and R analyses

`DistWin` projects contain not only the analyses but also the results of those analyses.  Analysis results are stored in `DistWin` projects according to a coding scheme to identify the statistics or parameter estimates.  Values of statistic and estimates can be extracted from a `DistWin` project using `get_stats()`; however this function is not intended for use by users.

Instead, `get_stats()` is usually called from within the function `test_stats()`.  The `test_stats()` function takes as its first argument a single analysis from a converted `DistWin` project.  The tasks performed by `test_stats()` are:

* run the specified analysis in `R`,
* extract the `DistWin`-generated results from the `DistWin` project and
* compare the two sets of results producing a table showing agreement (or disagreement) between `DistWin` and `R` results.

```{r, stratify-compare, comment=NA, echo=FALSE, warning=FALSE}
knitr::kable(test_stats(stratify.proj$'Half-normal cosine no stratification exact'))
```

The column labeled __Difference__ shows the _ratio_ of the `mrds` value over the `DistWin` value.  When called from the `R` console, the final __Pass__ column will possess ticks if the __ratio__ is less than 0.05 to quickly assess the agreement between `DistWin` and `mrds` result components.

Note `test_stats()` can take only a single analysis as an argument, rather than a complete set of analyses that may be embedded within a `DistWin` project.  To test multiple analyses, `lapply()` could be wrapped around the call to `test_stats()`.

There are a large number of reasons for disagreement between `DistWin` and `R` results.  Reasons may include differences in:

* convergence between optimisation engines in the two pieces of software,
* treatment of size-biased regression adjustment for objects detected in clusters.

# Migrations not performed by `readdst`

`readdst` is a work in progress.  As noted, it was developed for in-house testing of developing code against known solutions produced by `DistWin`.  There are many nuances of analyses that can be performed by `DistWin` that have not been included in `readdst` conversion feature.  The following list is incomplete, but gives a general idea of what is *not* included in `readdst` capabilities.

* multipliers
* bootstrapping of variances
* analyses involving the `dsm`, `mads` or `DSsim` engines
* post-stratification.

For this reason we recommend using the latest version of `readdst` from GitHub using the following commands:

```{r, eval=FALSE}
install.packages("devtools")
devtools::install_github("DistanceDevelopment/readdst")
```



[^environment]: Note that environments in R have rather different behaviour to other R objects -- if you modify them within a function, you modify the object everywhere (so called "reference semantics"). This can have unexpected consequences when you pass them to your own functions. See [http://adv-r.had.co.nz/Environments.html](http://adv-r.had.co.nz/Environments.html) for more information.
