Romancing the Open Source GIS Monster Jens Hegg Friday, February 13, 2015

Do you need to do GIS work but don't have the money for ArcGIS or the required extensions to do what you want to do? Have you heard about QGIS or GRASS and thought that they might solve your GIS woes with a free, open source solution? You might be right…or you might be very wrong. It all depends on what you hope to use them for and what your background is in programming languages.

I am a huge proponent of open source software but, before you start using a new open source package it is well worth your while to know how much time you are willing to commit. How far down the rabbit hole of coding and scripting you are prepared to go to get what you want? In some cases this isn't an issue and things work right out of the box, but it's worth it to know your limits ahead of time. This is particularly true for the open source GIS packages available out there.

I use multiple open source software packages and some are absolutely vital to what I do as a scientist. If there were one piece of software that I couldn't do without as an ecologist it would be R, the open-source statistical package that has become ubiquitous over the last few years. Open Office and specifically Libre Office have saved multiple files from Excel and Word that have been corrupted by their own office suite. I've been using Libre Office primarily for a couple months now.

Open source software promises a future where everyone can get what they need out of the software they use, rather than having features added only when it makes monetary sense for the company making it. It also provides, at least in theory, a very approachable and ever changing help system where questions can be answered by other users and problems can be reported and quickly fixed by those with the interest and skills to do it. The idea is great, and in the case of the programs I listed above it has worked out really well. But, for less developed software this isn't always the case.

I've recently been tearing my hair out trying to complete a GIS project that should have taken hours and instead has taken days. Because I'm a stubborn bastard who believes in open source (dammit!) and I didn't have an available license for ArcGIS and Arc Hydro Tools I have refused to back down, thinking that if I could only learn a bit about Python or a bit more about bash shells for Unix I could get done what I want to get done. It didn't work. What I wanted to to do is possible, but it's further down the rabbit hole than I have time or skills to go right now.

The interface of ArcMap 10.2, the industry standard GIS software

I should provide a bit of important background information. During my masters I published a paper showing that from relatively simple geologic maps, and some knowledge of the water chemistry in your study area, it is possible to predict what the strontium isotopic ratio will be anywhere along the river. This is hard to communicate over the thump of electronic music, and doesn't even go down well at refined cocktail parties. It's super interesting if you study where fish go using the isotopes in their ear bones, which is what I do as a PhD student.

Interface of ArcMap 10.2, the industry standard GIS software.

To do this work I used ArcGIS, the "industry standard" GIS platform. Personally, I dislike ArcGIS quite a bit (file names have to be less than 13 characters? This is 2015). So, for an upcoming paper on the migration patterns of giant Amazonian catfish I used a combination of QGIS and GRASS, two open source geospatial packages that inter-relate with each other and provide almost all of the functionality available in ArcGIS. Once I learned how to use them, these packages worked fine for fairly simple manipulation of maps, analysis of watersheds and geology, and creating really nice maps for publication.The vast majority of the tasks are automated and run through interactive windows like any other program. Where the problems cropped up was in trying to do anything more complicated.

Interface of QGIS 2.0. Notice the similarity to ArcMap.

QGIS in particular is a great platform that I think has tremendous promise. The interface is similar to ArcGIS but much more intuitive in my opinion. It accepts files of all kinds and is flexible about map projections. It also brings almost every other open source GIS resource into its toolbox, allowing you to use the tools available in all of them under one interface. I'm very excited about the future of QGIS, I think it will be the future. Just as R is supplanting expensive SAS licenses in academic publications I see QGIS doing the same thing to ArcGIS in the future.

Interface of GRASS 7.0 is very different, as is the functionality.

GRASS is a more picky beast, like a 70's muscle car, you can feel the power under the hood but the entertainment system and seats aren't quite as snazzy as a new Ferrari. If you don't know much about map projections GRASS will teach you like an angry stepmother, with beatings behind the woodshed until you get it. For those of us who have been coddled by ArcGIS or QGIS and their "on-the-fly" projection it can be hard to make sure each new GRASS import is in the correct projection and that the import parameters are correct to make a functioning file. Then, once those files are in GRASS they disappear into a database and can't just be copied around like a .shp file, they must be exported. Figuring out the difference between a "Location" and a "Mapset" (the names for the database locations and the organization within them) is not well explained. Further, if you require overlapping polygons in one file GRASS cannot do this, it uses a different type of vector representation that doesn't allow it. This attention to map projections and reliance on a database has an upside though. Once the maps are input you can be sure that the calculations you do aren't introducing errors because of the way the layers are projected.

Both of the software packages are fairly mature, but the way they evolved has made QGIS much more intuitive and user friendly. While QGIS seems to have focussed on usability and GUI development, GRASS grew out of a command line interface in UNIX and has only recently released a somewhat polished GUI. Much of it's functionality is still accessed through scripting. You have to drag a Mapset folder onto the app to start the GUI, a double click doesn't work on my Mac. The GUI starts as a wxPython icon via a terminal window…it isn't what you expect but it work. The script languages you must use seem to depend greatly on the operating system you use and the window you pick to enter scripts. For instance, one can enter commands directly into the terminal window in bash script (who's grammar varies it seems), or you can enter GRASS scripts into the command console of the GUI using a language that seems very similar to R, or you can enter Python scripts using the Python Shell that is integrated into the GUI…or you can use the GUI interface. Making sense of the online documentation with this mess of input languages is difficult at best.

That brings us to the factor that sets good open source software apart from the rest, the online documentation and the user community. Programs like Libre Office are so mature that the documentation is very similar to MS Office and anyone can get going with it right away. The R statistical package has a steeper learning curve but the user community is huge, and growing, so there is plenty of documentation. They also have a very strict documentation requirements for packages on the CRAN repository so that it's easy to figure out what each new add-on package does and how to use it (assuming you are familiar enough with statistics to understand the concepts being used). Also, R has a single input language so every command is intelligible. That isn't the case for the open source GIS packages yet.

QGIS is making a good start, with a polished website and some good documentation. But, if you are interested in anything beyond the basic functionality it can be really difficult to figure it out. The user community seems to be pretty small, and the program relies on integrating the functions of multiple other GIS packages…most of which I was unable to make work in my installation. Because each of these other packages is built on different code bases and uses different inputs, figuring out how to make add-ons work is really confusing.

GRASS has even more problems, in my opinion. The program itself is very powerful. Unfortunately, the way the program has been developed means that the range of input languages, and even the grammar of each, depends on your operating system as well as the add-ons you use within the program. The website itself does a terrible job of introducing this coding complexity. The user community compounds the problem by doing a terrible job of specifying in their posts what script languages they are using, and on what system it is meant to operate on. In R you can pick a snippet of script and play with it until it works. In GRASS you can't be sure that the example script you find is even runnable on your machine. It's incredibly frustrating unless you have a deep understanding of multiple scripting languages and the GRASS program itself. It is not for the faint of heart and this rabbit hole is deep.

So, if you are a poor graduate student looking at the money you could save using an open source GIS package here is my advice. If you are just doing fairly basic vector editing, overlay type analysis, or basic raster analysis then install QGIS and you won't be disappointed. If you need to do quite a bit of work with rasters, watershed analysis, or other terrain based analysis then GRASS will do the job. But, remember that neither of these packages do more complex analyses easily because you will need to script anything that requires repeated analysis of different features across a map. Some of this probably isn't hard if you have a background in Python. If you don't already have that background, and sometimes even if you do, then expect to spend a lot of time learning from mistakes and trying to piece together information from cryptic message board posts. I think the future is bright for open source GIS and in a few years I expect it will be very user friendly. Until then it isn't for everyone.

On Coffee and Science Education: sometimes a millimeter makes all the difference Jens Hegg Thursday, December 04, 2014

I'll tell you the moral to this story right off the bat. Sometimes a millimeter is the difference between something great and something terrible. Sometimes it pays to look at the details, understand why things work the way they do, and work through a problem logically. Sometimes it's fun to geek out on something, roll your pants into high waters, put in your pocket protector and act like an engineer. It might even save you twenty bucks. Sometimes I wonder if our country's lack of interest in science is partly due to our willingness to unthinkingly replace rather than understand and repair.

I recently bought a new coffee maker at World Market, a Pezzetti italian style moka pot. I did it because I thought the espresso machine I was given last year had kicked the bucket. It turns out that it's working fine again after apparently deciding to go on a short sabbatical for no reason I can determine. I was excited for the new moka pot because I just love the thick, almost syrupy coffee filled with deep caramel flavors that these little pots make. People confuse them with espresso makers but they really excel at the bottom end of coffee taste. Moka coffee usually lacks most of the high, bright, citrus notes in espresso. Both are good but they are very different animals. But I digress...

I filled the new pot with coffee and water and put it on the stove anticipating a stout cup of coffee. It boiled…and boiled…and boiled some more with steam coming out the top tube before it started spitting a tiny bit of extremely thick black coffee out of the collection spout. Then, after a few tablespoons it just quit and started making sputtering, spitting noises and smelling like the worst church-basement-percolated-through-a-two-hour-sermon brew you can imagine. That two tablespoons could have been mistaken for motor oil mixed with paint stripper. In a word it was undrinkable. I tried again a few days later and the result was the same.

I'm wasn't very familiar with these things, but the idea is pretty simple. Build up vapor pressure in the base, which forces the hot water up a siphon tube, through the grounds and into a collection chamber at the top. It's like a cross between a percolator and a pressure cooker. Take a look at this awesome video for a bit of geeky imagery of this in action filmed with neutron imaging. This isn't a complicated machine, so what was the problem?



I couldn't immediately see a problem when I took it apart. The seal and all the filters were fine. I ended up putting it away but I kept thinking, "this is too simple for it not to work when all the parts look right." My mind kept churning on it off-and-on before I realized what had to be happening. If no water was going up the tube then somehow it was losing the vapor pressure in the bottom chamber, since that is the force that sends the water upward. But how was that happening if I couldn't see or feel a leak on the outside?

Top rim is a millimeter or so higher than the filter basket

Then I remembered that steam had been coming out the top for quite awhile before any coffee came out. So, steam was somehow getting out around the filters inside. I went to look at the seal again and happened to notice that the top rim of the bottom half of the pot was slightly higher than the filter that fits inside it and holds the coffee. It was only a millimeter or a bit more, but perhaps that distance was too much for the rubber seal to bridge, letting steam out around the edge of the coffee basket. Hmm…

grinding down the rim on a water cooled lapping wheel

Over the next week I dropped into a couple of stores to take a look at their moka pots. Sure enough, all of them had rims on the bottom chamber that were the exact same height as the filter basket. It was time to experiment. I loaded up the coffee pot with water and grounds, then tightened it as hard as I could to, hopefully, create a better seal. It didn't really work, but I did get about twice as much motor oil/paint stripper solution out of the pot this time around so i figured a better seal must be helping.

Luckily, I had an easy way to fix the problem. I took the pot to my lab and ground the rim down with the sandpaper on a lapping wheel. You could do this easily enough with sandpaper mounted to a board, but a lapping wheel takes less time and the water jet helps keep things cool. You can see from the picture that the rim matches the height of the filter basket now after just a minute or two of grinding.

Rim even with filter basket after grinding

Then came the moment of truth. I loaded the filter basket with freshly ground Indonesia Gajah Aceh from Stumptown, filled the bottom chamber with water, and put it on the stove. This coffee is already just full of deep, intense, earthy flavors…my mouth was watering imagining how rich this coffee would be. I hoped I wasn't going to be disappointed yet again.

It worked! No steam came from the tube in the center of the collection pot, instead a steady stream of coffee issued forth with no sputtering or spitting until the pot was full and the water chamber was empty. The coffee was awesome…and is fueling this late night blogging session as we speak; thick, rich and chock full of bitter chocolate, carmel and straight up dirt.

Sometimes it pays to work through a problem like an engineer, understanding how things work and the principles behind it. I think too often now days we would rather buy something new than fix what we have and that is a shame. My grandfather used to spend lots of time tinkering and fixing things, and creative repair is a daily occurrence for my father the wheat farmer. Watching them taught me that if you understand how something works you can usually fix it or improve it. That is the essence of engineering and science.

Perhaps we Americans have so few kids interested in science because we aren't teaching them how rewarding it is to observe, troubleshoot, and creatively repair things in our lives. You can't learn that skill if you throw things away every time they don't work. Which reminds me…maybe I should figure out why that espresso machine decided to quit turning on for two weeks and then fixed itself for no reason. I'll keep thinking on that one while I try desperately to finish my PhD. Until then, my advice is to avoid moka pots from World Market unless you want to repeat what I just did.

The moment of truth. A perfect brew with no sputtering!

Guest post at Man Makes Fire Jens Hegg Wednesday, July 23, 2014

Gorgeous Cutthroat from a Tributary of Priest Lake     ©Jens Hegg

My friend Chris over at Man Makes Fire asked me to write a guest post on what it's like to use a tenkara style fly rod. It turned out to be a pretty awesome article that was a ton of fun to write. Check it out, along with the other stories, at Man Makes Fire - Fly Fishing with a Tenkara Rod

Top: 1oz, 7'10", Shimotsuke Kiyotaki rod
Middle: 3.1oz, 11'8", Shimotsuke 3.6 rod
Bottom: Reddington CT 4-piece, 5wt, handle section for comparison
© Jens Hegg