Romancing the Open Source GIS Monster

Do you need to do GIS work but don't have the money for ArcGIS or the required extensions to do what you want to do? Have you heard about QGIS or GRASS and thought that they might solve your GIS woes with a free, open source solution? You might be right…or you might be very wrong. It all depends on what you hope to use them for and what your background is in programming languages.

I am a huge proponent of open source software but, before you start using a new open source package it is well worth your while to know how much time you are willing to commit. How far down the rabbit hole of coding and scripting you are prepared to go to get what you want? In some cases this isn't an issue and things work right out of the box, but it's worth it to know your limits ahead of time. This is particularly true for the open source GIS packages available out there.

I use multiple open source software packages and some are absolutely vital to what I do as a scientist. If there were one piece of software that I couldn't do without as an ecologist it would be R, the open-source statistical package that has become ubiquitous over the last few years. Open Office and specifically Libre Office have saved multiple files from Excel and Word that have been corrupted by their own office suite. I've been using Libre Office primarily for a couple months now.

Open source software promises a future where everyone can get what they need out of the software they use, rather than having features added only when it makes monetary sense for the company making it. It also provides, at least in theory, a very approachable and ever changing help system where questions can be answered by other users and problems can be reported and quickly fixed by those with the interest and skills to do it. The idea is great, and in the case of the programs I listed above it has worked out really well. But, for less developed software this isn't always the case.

I've recently been tearing my hair out trying to complete a GIS project that should have taken hours and instead has taken days. Because I'm a stubborn bastard who believes in open source (dammit!) and I didn't have an available license for ArcGIS and Arc Hydro Tools I have refused to back down, thinking that if I could only learn a bit about Python or a bit more about bash shells for Unix I could get done what I want to get done. It didn't work. What I wanted to to do is possible, but it's further down the rabbit hole than I have time or skills to go right now.
The interface of ArcMap 10.2, the industry standard GIS software
I should provide a bit of important background information. During my masters I published a paper showing that from relatively simple geologic maps, and some knowledge of the water chemistry in your study area, it is possible to predict what the strontium isotopic ratio will be anywhere along the river. This is hard to communicate over the thump of electronic music, and doesn't even go down well at refined cocktail parties. It's super interesting if you study where fish go using the isotopes in their ear bones, which is what I do as a PhD student.
Interface of ArcMap 10.2, the industry standard GIS software.
To do this work I used ArcGIS, the "industry standard" GIS platform. Personally, I dislike ArcGIS quite a bit (file names have to be less than 13 characters? This is 2015). So, for an upcoming paper on the migration patterns of giant Amazonian catfish I used a combination of QGIS and GRASS, two open source geospatial packages that inter-relate with each other and provide almost all of the functionality available in ArcGIS. Once I learned how to use them, these packages worked fine for fairly simple manipulation of maps, analysis of watersheds and geology, and creating really nice maps for publication.The vast majority of the tasks are automated and run through interactive windows like any other program. Where the problems cropped up was in trying to do anything more complicated.
Interface of QGIS 2.0. Notice the similarity to ArcMap.
QGIS in particular is a great platform that I think has tremendous promise. The interface is similar to ArcGIS but much more intuitive in my opinion. It accepts files of all kinds and is flexible about map projections. It also brings almost every other open source GIS resource into its toolbox, allowing you to use the tools available in all of them under one interface. I'm very excited about the future of QGIS, I think it will be the future. Just as R is supplanting expensive SAS licenses in academic publications I see QGIS doing the same thing to ArcGIS in the future.
Interface of GRASS 7.0 is very different, as is the functionality.
GRASS is a more picky beast, like a 70's muscle car, you can feel the power under the hood but the entertainment system and seats aren't quite as snazzy as a new Ferrari. If you don't know much about map projections GRASS will teach you like an angry stepmother, with beatings behind the woodshed until you get it. For those of us who have been coddled by ArcGIS or QGIS and their "on-the-fly" projection it can be hard to make sure each new GRASS import is in the correct projection and that the import parameters are correct to make a functioning file. Then, once those files are in GRASS they disappear into a database and can't just be copied around like a .shp file, they must be exported. Figuring out the difference between a "Location" and a "Mapset" (the names for the database locations and the organization within them) is not well explained. Further, if you require overlapping polygons in one file GRASS cannot do this, it uses a different type of vector representation that doesn't allow it. This attention to map projections and reliance on a database has an upside though. Once the maps are input you can be sure that the calculations you do aren't introducing errors because of the way the layers are projected.

Both of the software packages are fairly mature, but the way they evolved has made QGIS much more intuitive and user friendly. While QGIS seems to have focussed on usability and GUI development, GRASS grew out of a command line interface in UNIX and has only recently released a somewhat polished GUI. Much of it's functionality is still accessed through scripting. You have to drag a Mapset folder onto the app to start the GUI, a double click doesn't work on my Mac. The GUI starts as a wxPython icon via a terminal window…it isn't what you expect but it work. The script languages you must use seem to depend greatly on the operating system you use and the window you pick to enter scripts. For instance, one can enter commands directly into the terminal window in bash script (who's grammar varies it seems), or you can enter GRASS scripts into the command console of the GUI using a language that seems very similar to R, or you can enter Python scripts using the Python Shell that is integrated into the GUI…or you can use the GUI interface. Making sense of the online documentation with this mess of input languages is difficult at best.

That brings us to the factor that sets good open source software apart from the rest, the online documentation and the user community. Programs like Libre Office are so mature that the documentation is very similar to MS Office and anyone can get going with it right away. The R statistical package has a steeper learning curve but the user community is huge, and growing, so there is plenty of documentation. They also have a very strict documentation requirements for packages on the CRAN repository so that it's easy to figure out what each new add-on package does and how to use it (assuming you are familiar enough with statistics to understand the concepts being used). Also, R has a single input language so every command is intelligible. That isn't the case for the open source GIS packages yet.

QGIS is making a good start, with a polished website and some good documentation. But, if you are interested in anything beyond the basic functionality it can be really difficult to figure it out. The user community seems to be pretty small, and the program relies on integrating the functions of multiple other GIS packages…most of which I was unable to make work in my installation. Because each of these other packages is built on different code bases and uses different inputs, figuring out how to make add-ons work is really confusing.

GRASS has even more problems, in my opinion. The program itself is very powerful. Unfortunately, the way the program has been developed means that the range of input languages, and even the grammar of each, depends on your operating system as well as the add-ons you use within the program. The website itself does a terrible job of introducing this coding complexity. The user community compounds the problem by doing a terrible job of specifying in their posts what script languages they are using, and on what system it is meant to operate on. In R you can pick a snippet of script and play with it until it works. In GRASS you can't be sure that the example script you find is even runnable on your machine. It's incredibly frustrating unless you have a deep understanding of multiple scripting languages and the GRASS program itself. It is not for the faint of heart and this rabbit hole is deep.

So, if you are a poor graduate student looking at the money you could save using an open source GIS package here is my advice. If you are just doing fairly basic vector editing, overlay type analysis, or basic raster analysis then install QGIS and you won't be disappointed. If you need to do quite a bit of work with rasters, watershed analysis, or other terrain based analysis then GRASS will do the job. But, remember that neither of these packages do more complex analyses easily because you will need to script anything that requires repeated analysis of different features across a map. Some of this probably isn't hard if you have a background in Python. If you don't already have that background, and sometimes even if you do, then expect to spend a lot of time learning from mistakes and trying to piece together information from cryptic message board posts. I think the future is bright for open source GIS and in a few years I expect it will be very user friendly. Until then it isn't for everyone.




Labels: , , , , , , , ,

Here's to vacations!


The life of a PhD student feels a lot like this picture sometimes. It's a slog, and sometimes you feel like you are getting nothing done at all. That is both the blessing and the curse of an open ended job where, in a lot of ways, I make my own schedule and get to determine what questions I ask. But when things feel like a total slog sometimes it helps to get away.

We took a vacation to Seattle to visit some of Katie's graduate school friends who were coming up from Texas to see the Emerald City. We picked up Berlyn from his mom's, where he has been visiting for the last month, walked around Pike Place Market, explored Bainbridge island, and generally had a good time. The weather was great and the mountains were out. The sunsets over the Olympics were awesome.

Now that the weekend is over it's back to the grind. A high school student is helping out in the lab for a couple of weeks and amazingly we got a ton of work done today. I feel like I lifted out of a midsummer slump and the mountains of work don't seem quite so horrible. Here's to vacations!



Labels: , , ,

Skiing with Berlyn

My lab group went to Silver Mountain last week to go skiing. What a blast. My advisor brought his kids so we had a gaggle of youngsters plying the slopes for the first time. Berlyn is a natural. Here is a video taken in the first 15 minutes of skiing, followed by a video from later on in the day. Sorry for the poor quality of my phone cam.





about 5 minutes after the first video was taken some skiers saw berlyn in mid run and gave him a thumbs up. Berlyn had the presence of mind to look at them and give a thumbs up while hurtling at toddler speed down the hill. The effort unbalanced him and he nearly wrecked but pulled it out into a sweeping spin that left him gliding the last 25 feet or so in a reverse snowplow. I know that I was not that coordinated when I was his age.

For more pictures of the mountain, and riding the worlds longest gondola to get there, check out my photobucket page here.

Labels: , , , , , , ,

Back from Whence Ye Came


Salmon are pretty incredible creatures. I stopped by Spalding Park near Lapwai, Idaho (officially known as Nez Perce National Historic Park and containing an excellent museum of Nez Perce art and historical pieces reclaimed with the help of Walt Disney's wife, a native of Spalding, ID) to see the spawning Coho salmon this past week. I've never seen salmon spawning so it was an exciting moment. Watching them makes you realize that life is circular. Ashes to ashes and all that, but not in a depressing way at all.

The picture above is a spawned out female that I found while peering off the bank at a female preparing her redd. She had already begun the next cycle, the nutrients she carried from the ocean helping to keep her spawning stream healthy and productive for the next generation.

Salmon prepare a redd, or nest, in the gravel of the stream bed into which they lay their eggs which are subsequently fertilized by the male and covered. Each species does it differently. Coho are adapted to more silty areas and their redd making entails sweeping away the silt with their tails, while other species may mound the gravel more or less and choose particular sized gravel and silt levels to suit their evolutionary taste. The picture to the right show a female coho preparing her redd by laying on her side and flapping her tail to clear the silt from the gravel. The next picture shows what the redd looks like from above. The oblong patch of lighter colored gravel on the right side of the picture next to the bank is her redd. Bonus points if you can find the salmon in the picture. She saw me on the bridge and moved off her redd into the darker and more camouflaging area of the stream.

Coho were declared extinct in the Snake River and only recently have been reintroduced by the Nez Perce tribe. Lapwai creek runs into the Clearwater river at the park which is located on the Nez Perce reservation. The Park is named after Henry Spalding, a missionary from the 1800's who's grave marker is shown to the right.The park encompasses the mission site as well as the historic cemetery.

While I was there I had a bit of an epiphany about life and death. I'm not sure if the salmon were the spur to my thinking or
not.
A member of one of my classes, which centers around cross disciplinary issues of salmon recovery on Lapwai creek, is a Nez Perce Tribal member. While visiting the park earlier he mentioned that a close friend had died and that he would be buried in Spalding cemetery. I watched the grave being dug and saw it again when I returned to see the salmon.

I noticed that rather than dig the grave with a backhoe friends and family came out, peeled back the sod, and dug the grave by hand.
Regardless of your personal convictions I think most people are horrified when they learn how we bury our dead. I can say that when I die I hope I am not filled with immutable fluids and placed in an impermeable casket within a concrete box (to keep the grave from sinking as I decompose), and placed in a hole dug by some anonymous and surly city employee who will likely throw his cigarette butt in the hole when he's done.

I can only hope that I surround myself with friends and family that would see the social value in coming together to dig my grave the way it aught to be dug, by hand. I can only dream that I would be laid down in a simple pine box or shroud so that, like the salmon, I can go back from whence I came.

Labels: , , , , , , , ,

Slowing Down for the Dog Days of Summer


Hello to all the regular readers as well as the new folks visiting from Tangled Bank this week. Things will likely be slow for a while here but it is for a good purpose, trust me.

I am moving this week to start graduate school in aquatic ecology at University of Idaho. I will be working with Dr. Brian Kennedy in the Water Resources Program on a Masters project in salmon otolith microchemistry and life history analysis.

The non-technical way of putting this is that I will be plucking out the ear bones of salmon and analyzing them much like you would the rings of a tree. Strontium isotopes are different for every river and stream due to changes in underlying geology. These isotopes are deposited in the "tree rings" of the fishes ear bones. With this knowledge in mind you can decipher where a fish has been along its migration route over it's entire lifespan. Further analysis of the ear bone can give you the fish's age, allowing you to know with amazing accuracy the date that a fish was in a particular river or stream.

This information is vital to understanding the dynamics of a species' migration and how this is effected by dams, fishing, global warming, genetics, and any number of other factors. I hope to expand the research in salmon as well as expand the technique to other migratory species about which we know very little.

With the start of school I am considering turning this blog into more of a science/"trials and tribulations of a grad" student blog. We will see in what direction I head, but I think no matter where it goes it will be quite a ride. Hang on!

Wish me luck! Moving across the country with a three year old is not an easy task. I will keep you updated on my progress as time allows.

Labels: , , ,