Tidy line of 2D array plots each with a colorbar

This is something I keep coming back to and first posted about a while ago here.

Let’s say we have three 2D arrays - we want to plot them in a line, each with a colorbar that is the... </div> </article>

EGU 2019 - Maximising information acquisition in a world of ever increasing data availability

EGU 2019: Call for abstracts!

We draw your attention to the multidisciplinary session “Maximising information acquisition in a world of ever increasing data availability” (NP4.4/BG1.23/CR7.5/ESSI3.6/GM2.12/NH11.12/SSS11.8) to be... </div> </article>

List comprehensions - some examples (Python 3.x)

I’ve recently been using list comprehensions more often than not, having never paid them much attention. A decent overview is available here. Perhaps they are a sprinkling of syntactic sugar but I think I’m a fan.

Here are... </div> </article>

ENVI binary to GeoTiff - opening, processing and output in Python

Based on a post from a few years ago, a few people have asked how to handle ENVI format files in python for processing with the end goal being to output them as GeoTiff files. Purely for file format transformation,... </div> </article>

Create a presence absence matrix from long data (R)

Using R, I needed to convert some long format data to wide format in the form of a presence/absence dataset. The why doesn’t really matter :)

Where the initial dataset is for example a vector of categories, the intended... </div> </article>

Create random points based on a regular grid

Aim: Create 100 random points based on the cell centre coordinates of a regular grid

Using R, the way we’ll go about it is:

  1. Create a grid using the raster package to a given extent
  2. Extract the cell... </div> </article>

    Replicating GMT's blockmedian/blockmean in R

    I have frequently used GMT’s blockmean and blockmedian functions with great success. These method essentially quasi-grid your points which is a particularly useful technique when trying to reduce irregularly spaced data such as for onward point sampling... </div> </article>

    Comparing input and output files

    When running jobs on multiple files, sometimes you might find that files are missed. To ensure that you process ALL files, you can compare your total input files with those already processed using grep. Here’s an example of implementing... </div> </article>

    QGIS - handy plugins

    I was asked to demonstrate some plugins I find useful in QGIS. Some are pretty basic but are very handy! Here’s a list and associated summary:

    • Value tool: reports values of rasters beneath cursor
    • Zoom to Point: zooms to... </div> </article>

      Running a parallel loop in R

      A code snippet detailing how to run a loop in parallel in R. The code below creates multiple files (each with a random number in the title) with the text “testing 1,2,3…” written inside. Note that parellelisation has its own... </div> </article>

      Removing a component of a signal using FFT

      A component of a signal can easily be removed by using the Fast Fourier Transform (and its inverse) - in Python, this is easily implemented using numpy. The code below zeros out parts of the FFT - this should be... </div> </article>

      Add multiple csv files to QGIS quickly...

      I often find myself with csv files that I’d like to visualise quickly in QGIS. If these were shape files, it is quick and easy to load multiple files at once. This isn’t the case with csv files as you... </div> </article>

      Creating topography - synthetic fjords

      We recently published a paper on developing synthetic fjord bathymetry in the absence of observations. Before giving a summary of the paper, lets cover some background which may answer questions such as “why is synthetic bathymetry required”, “why not... </div> </article>

      Create a discrete colour bar (Python)

      When making maps or plots of spatial data (or any data really), colour bars are an almost standard requirement. For the map making process, I tend to create my basic map in ArcMap or QGIS, touching it up... </div> </article>

      Creating a DEM from regularly / irregularly spaced points (R and Python)

      DEMs (raster format) are created from point elevation observations. When working with a DEM, it is important to be aware that the values of a given cell are the result of some processing step that converted point elevations to a... </div> </article>

      Centre a diverging colorbar at a defined value with matplotlib

      With raster datasets, I often find myself using diverging colour scales. For a dataset ranging from say -3000 to 1000, we might want a colorbar to diverge from 0. By default though, any colorbar applied in matplotlib will diverge from... </div> </article>

      Matplotlib - getting to grips with colorbars and equivalent space

      You want to make two plots of two equally sized 2D arrays - one of which has a colorbar, the other not. You also want to ensure that both plots show the arrays at the same equivalent size. This requires... </div> </article>

      Converting csv files to ESRI Shapefile format

      The ability to visualise csv files within a GIS usually requires a couple of steps if you are operating your GIS in a point and click fashion. This is fine for a couple of files, but where you have multiple... </div> </article>

      A free book covering Python for spatial data processing

      It has come to my attention that there is a free work book available taking the reader from basic python, to general statistical analysis, plotting and ultimately spatial data processing (think rasters and remote sensing). This is an incredibly useful... </div> </article>

      Modify the appearance of your putty terminal

      Putty is an open source SSH client for windows. Various tutorials on how to make use of it exist (e.g. see here). On opening putty, you have a variety of options which you can make use of, including... </div> </article>

      Binary image skeletonisation with Python

      I’ve been looking to do some skeletonisation using python libraries and found something that did precisely that using the third party python package scikit-image. It was so quick to implement (after having spent a while trying to roll... </div> </article>

      Changing your Python version with Anaconda

      Python 3 is the future and the future is now. Considering best practise, the way forwards is to move with the times and upgrade. To make the change over easier, here’s a cheat sheet for writing python 2/3 compatible... </div> </article>

      Calculate a parabola from three known points

      I was looking for a quick fix for calculating values along a parabola given three known points. Scouting around I found a nice symbolic C function doing exactly what I wanted (see here). The function doing the work of... </div> </article>

      3D views of the Globe

      I was after a 3D projection of the globe focusing on some areas I had been working - I was after a horizon type view of the globe and so steps in the orthographic projection. Really, I was after a... </div> </article>

      Calling gdal utilities from within python using subprocess

      I am always using gdal commands in python. Sometimes though there are things that just happen faster though using the OSgeo4W Shell (I’m using windows at the moment) such as gdal_translate. It is still possible to use these commands... </div> </article>

      Reprojecting shapefiles

      I’m often using the gdal library for reprojecting and resampling raster data but have rarely needed to do so with vector formats. When dealing with points, I usually sort out the conversion manually. For other vectors - polygons and... </div> </article>

      Install natgrid add on for matplotlib

      Trying to use the nearest neighbour interpolation routine through matplotlibs griddata function, I realised I needed to make use of the natgrid extension package.

      Working on windows and after as simple a point and click approach as possible,... </div> </article>

      Resample raster to the size of another

      The usable code is the last snippet.

      I needed to resample and resize one larger array (call it arr_src) to the extent and post of a smaller array (arr_trgt). Using a mesh grid approach, I initially looped through the... </div> </article>

      Before I forget - logical_and from Numpy

      I keep coming back to needing this function, each time forgetting where to find it - populating an array based on conditions of other arrays which act as an input.

      > np.logical_and()

      Fixing errors associated with R's ncdf "get" function

      Reading in data from netcdf files can be a bit of a pain the first few times you have to deal with them - this is rarely helped by error occurrences that mean next to nothing to you. Whilst trying... </div> </article>

      Reading in NetCDF data in R and exporting as a geotiff

      This definitely passes my “post things that took more than half an hour to work out” as I know Aaron O’Leary was originally basing his posts on.

      I’ve never had to deal with netCDF files before. They are a... </div> </article>

      Co-ordinate systems - Projecting data in IDL

      There are three key commands you need to know about in IDL for projecting data

      1. map_proj_init
      2. map_proj_forward
      3. map_proj_inverse


      This sets the projection parameters required i.e. all details associated with the map projection The projections are represented... </div> </article>

      Bulk data downloads using FTP

      I’ve been needing to download vast amounts of data from various source recently including from the NSIDC. As with the NSIDC, there are various ways made available to you to download data. One useful area is the http file... </div> </article>

      Learn Python - learn to fly

      If ever you find yourself stuck with something, just think - you could probably do it more easily using Python:

      "Learn to fly" boundary - ideal

      Here is a great place to start... </div> </article>

      Calculating the concave hull of a point data set (Python and R)

      Following the calculation of a convex hull as described a few weeks ago, I’ve worked up a way to approximate a “concave” hull. This can be useful for point clouds of complicated geometries. Whereas the convex hull is a... </div> </article>

      Installing gdal binaries for Python on a windows machine

      The gdal library is an excellent source of tools that help you query, process and manipulate spatial data of varying formats. It can be used directly from the command line in both linux and windows environments and is also... </div> </article>

      ENVI binary files to numpy arrays using GDAL

      Most of my work currently revolves around raster processing of one kind or another. Being able to take advantage of the extensive libraries within Python (e.g. Numpy) is extremely helpful when carrying out such tasks. This blog really got... </div> </article>

      Snakes and ladders - rows and columns in Python

      Every time I have to work with rows and columns I always mix this up - it must be because of the “across the hall and up the stairs” phrase I remember from school geography lessons.

      When dealing with arrays... </div> </article>

      Problems of parallel processing - shared file names

      Ensure any temp files that may be written to don’t run the risk of being accessed by scripts being run in parallel - this can result in confusing results - especially if your program doesn’t crash!

      I found a handy... </div> </article>

      Calculating the convex hull of a point data set (Python)

      Working with LiDAR point data it was necessary for me to polygonize the point cloud extent. A first approach was to calculate the convex hull of the points. This is predominantly facilitated using scipy spatial’s ConvexHull function.

      For my... </div> </article>

      Resampling arrays and dataframes

      I’ve been experimenting with various different tools recently and for speed required smaller but representative data from my current project. Here are two ways of resampling an array by using only every nth line of the original array (python) or... </div> </article> </div>