Moving window operations across arrays

Applying functions using moving windows are a common feature of geospatial data analysis and often exist “under the hood” of GIS software. If you’ve ever calculated a slope or aspect grid from a raster elevation dataset, then the program you’ve... </div> </article>

Functions in functions with varying numbers of arguments

Developing a moving window operation framework to work on large numpy arrays (of digital elevation models), I need to modify one “master” moving window function to take in additional functions for which the number of variables might vary.

This master... </div> </article>

Joining vector datasets with duplicate field names with arcpy's fieldmappings object

I’ve been looking for how to spatially join vector data using an older version of ArcGIS (v10.3) where the data being joined may contain duplicate attribute names. ArcGIS’s python library arcpy provides access to the fieldmappings object... </div> </article>

Aspect values to categories

When dealing with terrain aspect values, you might find yourself needing to categorize them into sectors e.g. 10 deg N = North, 24 deg North = North-east, 90 deg North = East etc. This is also something to consider when... </div> </article>

Print to file from standard output using Python 3

To print standard output to a file in python 3, you can make use of the redirect_stdout function from the contextlib package. The following example uses this library to write the standard output printed from a... </div> </article>

Thinning your data out with pandas

When compiling large datasets, you may end up with values in some columns that make up only a small contribution of the total. These values may be important and should not just be discarded without careful consideration. Equally, if they... </div> </article>

Applying a function to all rows of a column with pandas

A common operation to implement with a pandas dataframe is to run a function for each entry or row of a column. Below shows how we can do this using a simple custom function.

First we’ll put together a dataframe... </div> </article>

Calculating time elapsed using timestamp information in pandas

Given a set of dates and times, we want to calculate the time elapsed of each row relative to the first entry. A csv file containing dates and times is available for you to download here. It looks like... </div> </article>

Tidy line of 2D array plots each with a colorbar

This is something I keep coming back to and first posted about a while ago here.

Let’s say we have three 2D arrays - we want to plot them in a line, each with a colorbar that is the... </div> </article>

EGU 2019 - Maximising information acquisition in a world of ever increasing data availability

EGU 2019: Call for abstracts!

We draw your attention to the multidisciplinary session “Maximising information acquisition in a world of ever increasing data availability” (NP4.4/BG1.23/CR7.5/ESSI3.6/GM2.12/NH11.12/SSS11.8) to be... </div> </article>

List comprehensions - some examples (Python 3.x)

I’ve recently been using list comprehensions more often than not, having never paid them much attention. A decent overview is available here. Perhaps they are a sprinkling of syntactic sugar but I think I’m a fan.

Here are... </div> </article>

ENVI binary to GeoTiff - opening, processing and output in Python

Based on a post from a few years ago, a few people have asked how to handle ENVI format files in python for processing with the end goal being to output them as GeoTiff files. Purely for file format transformation,... </div> </article>

Create a presence absence matrix from long data (R)

Using R, I needed to convert some long format data to wide format in the form of a presence/absence dataset. The why doesn’t really matter :)

Where the initial dataset is for example a vector of categories, the intended... </div> </article>

Create random points based on a regular grid

Aim: Create 100 random points based on the cell centre coordinates of a regular grid

Using R, the way we’ll go about it is:

  1. Create a grid using the raster package to a given extent
  2. Extract the... </div> </article>

    Replicating GMT's blockmedian/blockmean in R

    I have frequently used GMT’s blockmean and blockmedian functions with great success. These method essentially quasi-grid your points which is a particularly useful technique when trying to reduce irregularly spaced data such as for onward point sampling... </div> </article>

    Comparing input and output files

    When running jobs on multiple files, sometimes you might find that files are missed. To ensure that you process ALL files, you can compare your total input files with those already processed using grep. Here’s an example of... </div> </article>

    Running a parallel loop in R

    A code snippet detailing how to run a loop in parallel in R. The code below creates multiple files (each with a random number in the title) with the text “testing 1,2,3…” written inside. Note that parellelisation has its own... </div> </article>

    Removing a component of a signal using FFT

    A component of a signal can easily be removed by using the Fast Fourier Transform (and its inverse) - in Python, this is easily implemented using numpy. The code below zeros out parts of the FFT - this should be... </div> </article>

    Add multiple csv files to QGIS quickly...

    I often find myself with csv files that I’d like to visualise quickly in QGIS. If these were shape files, it is quick and easy to load multiple files at once. This isn’t the case with csv files as you... </div> </article>

    Creating topography - synthetic fjords

    We recently published a paper on developing synthetic fjord bathymetry in the absence of observations. Before giving a summary of the paper, lets cover some background which may answer questions such as “why is synthetic bathymetry required”, “why not... </div> </article>

    Create a discrete colour bar (Python)

    When making maps or plots of spatial data (or any data really), colour bars are an almost standard requirement. For the map making process, I tend to create my basic map in ArcMap or QGIS, touching it up... </div> </article>

    Creating a DEM from regularly / irregularly spaced points (R and Python)

    DEMs (raster format) are created from point elevation observations. When working with a DEM, it is important to be aware that the values of a given cell are the result of some processing step that converted point elevations to a... </div> </article>

    Centre a diverging colorbar at a defined value with matplotlib

    With raster datasets, I often find myself using diverging colour scales. For a dataset ranging from say -3000 to 1000, we might want a colorbar to diverge from 0. By default though, any colorbar applied in matplotlib will diverge from... </div> </article>

    Matplotlib - getting to grips with colorbars and equivalent space

    You want to make two plots of two equally sized 2D arrays - one of which has a colorbar, the other not. You also want to ensure that both plots show the arrays at the same equivalent size. This requires... </div> </article>

    Converting csv files to ESRI Shapefile format

    The ability to visualise csv files within a GIS usually requires a couple of steps if you are operating your GIS in a point and click fashion. This is fine for a couple of files, but where you have multiple... </div> </article>

    A free book covering Python for spatial data processing

    It has come to my attention that there is a free work book available taking the reader from basic python, to general statistical analysis, plotting and ultimately spatial data processing (think rasters and remote sensing). This is an incredibly useful... </div> </article>

    Modify the appearance of your putty terminal

    Putty is an open source SSH client for windows. Various tutorials on how to make use of it exist (e.g. see here). On opening putty, you have a variety of options which you can make use of, including... </div> </article>

    Binary image skeletonisation with Python

    I’ve been looking to do some skeletonisation using python libraries and found something that did precisely that using the third party python package scikit-image. It was so quick to implement (after having spent a while trying to roll... </div> </article>

    Changing your Python version with Anaconda

    Python 3 is the future and the future is now. Considering best practise, the way forwards is to move with the times and upgrade. To make the change over easier, here’s a cheat sheet for writing python 2/3 compatible... </div> </article>

    Calculate a parabola from three known points

    I was looking for a quick fix for calculating values along a parabola given three known points. Scouting around I found a nice symbolic C function doing exactly what I wanted (see here). The function doing the work of... </div> </article>

    3D views of the Globe

    I was after a 3D projection of the globe focusing on some areas I had been working - I was after a horizon type view of the globe and so steps in the orthographic projection. Really, I was after a... </div> </article>

    Calling gdal utilities from within python using subprocess

    I am always using gdal commands in python. Sometimes though there are things that just happen faster though using the OSgeo4W Shell (I’m using windows at the moment) such as gdal_translate. It is still possible to use these commands... </div> </article>

    Reprojecting shapefiles

    I’m often using the gdal library for reprojecting and resampling raster data but have rarely needed to do so with vector formats. When dealing with points, I usually sort out the conversion manually. For other vectors - polygons and... </div> </article>

    Install natgrid add on for matplotlib

    Trying to use the nearest neighbour interpolation routine through matplotlibs griddata function, I realised I needed to make use of the natgrid extension package.

    Working on windows and after as simple a point and click approach as possible,... </div> </article>

    Resample raster to the size of another

    The usable code is the last snippet.

    I needed to resample and resize one larger array (call it arr_src) to the extent and post of a smaller array (arr_trgt). Using a mesh grid approach, I initially looped through the... </div> </article>

    Before I forget - logical_and from Numpy

    I keep coming back to needing this function, each time forgetting where to find it - populating an array based on conditions of other arrays which act as an input.

    > np.logical_and()

    Fixing errors associated with R's ncdf "get" function

    Reading in data from netcdf files can be a bit of a pain the first few times you have to deal with them - this is rarely helped by error occurrences that mean next to nothing to you. Whilst trying... </div> </article>

    Reading in NetCDF data in R and exporting as a geotiff

    This definitely passes my “post things that took more than half an hour to work out” as I know Aaron O’Leary was originally basing his posts on.

    I’ve never had to deal with netCDF files before. They are a... </div> </article>

    Co-ordinate systems - Projecting data in IDL

    There are three key commands you need to know about in IDL for projecting data

    1. map_proj_init
    2. map_proj_forward
    3. map_proj_inverse


    This sets the projection parameters required i.e. all details associated with the map projection The projections are represented... </div> </article>

    Bulk data downloads using FTP

    I’ve been needing to download vast amounts of data from various source recently including from the NSIDC. As with the NSIDC, there are various ways made available to you to download data. One useful area is the http file... </div> </article>

    Learn Python - learn to fly

    If ever you find yourself stuck with something, just think - you could probably do it more easily using Python:

    "Learn to fly" boundary - ideal

    Here is a great place to start... </div> </article>

    Calculating the concave hull of a point data set (Python and R)

    Following the calculation of a convex hull as described a few weeks ago, I’ve worked up a way to approximate a “concave” hull. This can be useful for point clouds of complicated geometries. Whereas the convex hull is a... </div> </article>

    Installing gdal binaries for Python on a windows machine

    The gdal library is an excellent source of tools that help you query, process and manipulate spatial data of varying formats. It can be used directly from the command line in both linux and windows environments and is also... </div> </article>

    ENVI binary files to numpy arrays using GDAL

    Most of my work currently revolves around raster processing of one kind or another. Being able to take advantage of the extensive libraries within Python (e.g. Numpy) is extremely helpful when carrying out such tasks. This blog really got... </div> </article>

    Snakes and ladders - rows and columns in Python

    Every time I have to work with rows and columns I always mix this up - it must be because of the “across the hall and up the stairs” phrase I remember from school geography lessons.

    When dealing with arrays... </div> </article>

    Problems of parallel processing - shared file names

    Ensure any temp files that may be written to don’t run the risk of being accessed by scripts being run in parallel - this can result in confusing results - especially if your program doesn’t crash!

    I found a handy... </div> </article>

    Calculating the convex hull of a point data set (Python)

    Working with LiDAR point data it was necessary for me to polygonize the point cloud extent. A first approach was to calculate the convex hull of the points. This is predominantly facilitated using scipy spatial’s ConvexHull function.

    For my... </div> </article>

    Resampling arrays and dataframes

    I’ve been experimenting with various different tools recently and for speed required smaller but representative data from my current project. Here are two ways of resampling an array by using only every nth line of the original array (python) or... </div> </article> </div>