Monthly Archives: November 2012

Trying out the d3 matrix plugin by Erik Solen

I missed a d3 meetup event this fall (due to a hurricane Sandy related scheduling adjustment). I heard great things about Erik’s talk and I set myself an action item of taking a look at what he presented, wish I had been there…

Turns out the talk was focused on making a jQuery plugin of some d3 code. As I have yet to architect any complicated sites, a lot of this was well over my head, especially without the audio of the talk. But I did some yak-shaving trying to understand what this code was about and what problems (which I have yet to run into) are being solved.

I got a rudimentary introduction to the following concepts, big and small ones:

  1. Require.js, AMD, “define”
  2. widget factory with jQueryUI, _create() versus _init() etc.
  3. d3 Nested Selection
  4. html table headings <thead>, yeah, I never used them before. :)
  5. unshift(), dito.
  6. XMLHttpRequest Error, can’t get to local file “url” with file request, must use HTTP request
  7. Python SimpleHTTPServer, serves up a local file through a GET request.
  8. “json” format uses double quotes only (prepared data file is read as ‘json’ and this threw me as default python output was with single quoted strings). No useful errors, where woud I have found them?  I guess this is more confusing because the file has a .js extension?
  9. buster javascript unit testing
  10. Fun emacs tricks:
  11. Ajax…

Questions that I have that are unanswered:

  1. What is a good way to get your data from python to javascript? I keep writing python code to output valid javascript, which seems pretty similar to what Erik did too. Is there a better practice?
  2. Why use ajax? Requires the HTTP server running, etc…
  3. Why not put the table sideLabels and topLabels in with the data? Humm… in this case it made sense to me so I did, though I guess if they weren’t changing it might make sense to have them separate.

Unfortunately I am too much a novice to understand the real value of this talk without the talk. I did use the code though to plot the adjacency matrix of character interactions from Shakespeare’s Antony and Cleopatra, which I was performing with friends on the night of the talk. This is not the best application for the d3 matrix plugin, but I guess I gleaned what I could from this missed opportunity.

Since what I really wanted is a matrix, it would be nice if it were square and the topLabels vertical, in which case it really makes more sense to use svg than to use a table. So while not perfect, I figured this was a good enough stopping point for this investigation.

Using Mapnik to render custom colored Map images

Say I want my output map of the world to be colored with a customer specified set of specific colors for countries, water etc.  I also want to output a specific resolution for publication.

Map of World rendered in Mapnik

Natural Earth shapfiles have a the built in “mapcolor” attribute to group countries appropriately for this purpose, that will be helpful!

Q-GIS has some nice functionality for playing around with the shapefile and exploring the data. But for a final output image it doesn’t meet the following requirements:

  • Controlling output image resolution and cropping (you can save to .png or .jpg, but there are no other options regarding resolution or dimensions to save).
  • Configuring colors and color ranges to use can be done (quick and dirty demonstration that the mapcolors attribute is what I expect is easy). But using Q-GIS quickly becomes a super manual process, you have to specify colors via separate RGB fields, ugh. You can specify an attribute to use for the color categories, but seems hard in that mode to configure specific colors to use.
  • Must reload modified shapefiles and re-configure them anytime the source file changes.

I don’t doubt that there are plugins for saving files (say if you want a vector image!!) and I could probably create my own symbology configuration for the colors I want, but since I am only catering to myself and don’t want to become an expert in some RSI inducing graphical interface… time to find a scriptable way to generate high quality output images.

Mapnik is a map renderer that has bindings for lots of languages including python. I started with Getting started with mapnik in python, which walks through a simple example in no time. First problem solved: simply specify the image size when you declare the map:

m = mapnik.Map(600,300)

And specify the area you want to display when you save it:

m.zoom_to_box(mapnik.Box2d(-180, -90, 180, 90)) # or m.zoom_all()  
mapnik.render_to_file(m,'world.png', 'png')

Unfortunately, coloring the country groups is not trivial. Mapnik doesn’t current have the ability to set the PolygonSymbolizer’s fill color based on a shapefile attribute (feature in the works). However, as I am now an expert at manipulating shapefiles with pyshp, I hacked together an ugly but straightforward solution in a few minutes.

  1. Using pyshp, read in the shapefile to render and spit out n new files, one for each mapcolor group. See
  2. Create a unique mapnik style for each country color group and add each new shapefile to the map with the appropriate style layer. See
Putting it all together, I used the script and other utility functions to generate the colored map of the world above. 

Using the same techniques morphed shapefiles can be rendered in the same color pallet, change your mind about  the colors? No problem.

Generating a cartogram of access to pain medication around the world.

Okay! Now that I have my data in a shiny new shapefile. Time to make some cartograms using ScapeToad.

The Data:  Morphine per Death as reported by the Global Access to Pain Relief Initiative (GAPRI)

I am working on this project with Kim Ducharme for WGBH, The World, for a series on cancer and access to pain medication, which highlights vast disparities in access between the developed and developing world. Below is a snippet of the data obtained from GAPRI, showing the top 15 countries for amount of morphine available/used per death by Cancer and HIV and the bottom 15 for which there were data.

                                   Country   mg Morphine/ Death
                            --------------   -------------------
                             United States   348591.575146
                                    Canada   332039.707309
                               Switzerland   203241.090828
                                   Austria   180917.511535
                                 Australia   177114.495731
                                   Denmark   160465.864664
                Iran (Islamic Republic Of)   149739.130818
                                   Germany   144303.589803
                                   Ireland   140837.280443
                                 Mauritius   121212.701934
                            United Kingdom   118557.183885
                                     Spain   116480.684253
                               New Zealand   112898.731957
                                   Belgium   108881.848319
                                    Norway   106706.195632

And the 15 countries with the least amount of morphine access:

                                   Country   mg Morphine/ Death
                            --------------   -------------------
                                   Burundi       38.261986
                                  Zimbabwe       34.508702
                                     Niger       31.359717
                                    Angola       30.485112
                                   Lesotho       25.998371
                                  Ethiopia       25.323131
                                      Mali       24.713729
                                    Rwanda       23.269946
                                  Cameroon       15.162560
                                      Chad       10.866740
                             Côte D'Ivoire        9.723552
                                  Botswana        9.352994
                                   Nigeria        8.780894
                              Sierra Leone        8.546830
                              Burkina Faso        7.885819

Traditional Cartogram

Based on these numbers of morphine/death, in a basic cartogram where each country’s area becomes proportional to the metric, Switzerland would be 60% of the size of the US. But wait… this wasn’t what I expected, gosh that’s ugly and hard to read… And so starts the cartogram study and tweaking experiment. Is there a perfect solution?

Morphine per Death (as Mass)

Note the countries of Europe are too constrained to get to their desired sizes, so there is always some error in these images. Regardless of that there are two issues: 1) Europe/Africa/Asia are so badly distorted as become nearly unreadable, and bring the emphasis to a fish-eye view of Europe with weird France and Switzerland shapes. 2) This seems to make the whole story about Europe, de-emphasizing the US and Canada, which have higher usage than any of the European countries and also taking the focus away from shrunken Africa/Asia and South America.

This seems to be the best that the diffusion based contiguous cartogram is going to be able to do for this data set. ScapeToad has some options for mesh size and algorithm iterations, none of which seem to significantly effect the output image in this case. The other option is to take your metric and apply it as a “Mass” (as above) or as a “Density” to each shape. ScapeToad explains what the Mass/Density distinction is pretty well:

In our case Morphine/death is a “Mass/Mass” ratio which is also a “Mass”. However, for kicks I ran the “Density” option which is technically wrong (scales the area of each country based on the metric, instead of making the area proportional to the metric as a traditional cartogram should). Low and behold, the density image is certainly more satisfying and seems to tell a better story, although over-emphasizing the role of the US, Canada and Australia, which all dwarf Europe:

Morphine per Death (as Density)

Well, this is a quandary, the “correct” image is too confusing to be useful and takes the focus away from the story about the developing world and into what-the-?-is-this-distorted-picture land. But the “density” image is not “correct”.

From here I spent some time trying to generate a less distorted mass based cartogram. By running the cartogram generation on each continent separately I generated much less distorted images of Europe and Africa (Asia still needs some work). Shown here are the raw outputs for these regions in green, purple and pale blue respectively.

Morphine per Death (as Mass) by region, unscaled

To piece the cartogram back together the continents needed to be scaled and translated to the correct locations. Here is how far I got in that process. Europe is much easier to read and Africa is a huge improvement. Asia/the Middle East are still quite confusing, potential for improvement breaking this into more chunks, but it was becoming a more and more manual process and the output image still isn’t “satisfying”.

Morphine per Death (as Mass) each region calculated separately, then scaled appropriately to maintain more recognizable shapes

Does this cartogram tell the story we want? Does it really make sense to honor country borders and make small countries as large as big countries that have the same morphine/death value? For example, all things remaining equal, if the German and French speaking parts of Switzerland split into two new countries, given the same morphine/death number should each of the two halves have a cartogram area equal to previous Switzerland, effectively doubling the size because of  a political change? That doesn’t make much sense, but that would be considered a technically “correct” cartogram measure. It seems to me in some ways scaling the area is more correct as in the Morphine/Death as Density image, as it doesn’t exaggerate small countries with smaller populations…

In the end it is possible to generate a lot of different cartogram images. Some of which are suggestive of the story you want to tell, none of which are easily deciphered to provide actual data numbers. Keeping in mind that a cartogram isn’t a tool for communicating precise data measures, I think pick the one you like that makes sense to you vis-á-vis your data and the story you want to tell, don’t overstate the accuracy of the image, and provide other means to get at the actual numbers. For example, I created an alternate view in this choropleth map of the same data.

UPDATE 2012/12/03:

The final image is published now as part of PRI’s The World new series on Cancer’s New Battleground — The Developing World.

Access to Pain Medication around the World

Adding custom data to a shapefile using pyshp 1.1.4

As part of a cartogram generating project I need to get data from a .xls file into a shapefile for use with ScapeToad. Reading the excel file is easy with xlrd. Shapefiles? that is new territory.

Shapefile resources:

ScapeToad requires a high quality shapefile as input. I tested two that worked fine, both include population data suitable for testing:

  1. Thematic Mapping: Which is licensed under the Create Commons Attribution-Share Alike License. This license is not appropriate for commercial use and the author didn’t respond to my question regarding other licensing options.
  2. Natural Earth: On a tip from Andy Woodruff, I switched to using Natural Earth shapefiles which are licensed completely in the public domain, suitable for anything you want! Note, I discovered that the most zoomed out Natural Earth file, “ne_110m”, didn’t have shapes for some of the smaller island countries in my data set, so switched to using the “ne_50m” versions which included everything I needed.
Next step, getting custom data into the shapefile.

Using pyshp to add data to a shapefile

Since I do most of my data processing in python anymore, I was happy to find a python module for read/write/editing shapefiles. Unfortunately, is not the best maintained module. I used pyshp version 1.1.4, because it was packaged in ubuntu. After discovering a number of bugs I realized they have already been reported but nothing significant seems to have been fixed in 1.1.6. So I will just document the workarounds I used here.

1st pyshp 1.1.4 workaround: Renamed the shapefiles to remove any dots in the file name (the 0.3 in the case of thematic mapping shapefiles) because pyshp can’t handle extra dots in the file name.

This is kinda a nuisance since there are 4 files in a “shapefile”. This command will rename the extensions “dbf”, “prj”, “shp” and “shx” all at once:

 for a in dbf prj shp shx;do mv TM_WORLD_BORDERS-0.3.$a TM_WORLD_BORDERS_dot_free.$a;done

2nd pyshp 1.1.4 workaround: Massage numeric data you are adding to a record to have the correct precision.

My whole reason for using pyshp is to add data from excel into the shapefile. This means adding fields to identify the data and then adding the data to the record for each shape. The format of the new attributes (a.k.a. fields) is well described here. In my case I want to add numbers for example: sh.field(‘MY_DATA’, ‘N’, 6, 3). The number args are width and precision, where width is the total number of characters to represent the number and precision is the number of characters after the decimal. The above (6,3) can encode: -1.234 and 98.765.

Note, pyshp will error (AssertionError assert len(value) == size) if you put data into the record with greater precision than specified (it will not truncate for you).  I used a simple hack below to get a precision of 3 for my data:

    def precision3(n):
        ''' Force argument to precision of 3'''
        return float('%0.3f'%n)

3rd pyshp 1.1.4 workaround: When adding a new data field, pad all the records with default data for the new attribute.

pyshp assumes when saving the file that the data is perfectly formatted, but doesn’t help too much when adding or deleting data. Records are stored in python as a list of lists, when the shapefile is written pyshp assumes that the records lengths equal the number of fields (as they should be). But it is your job to make sure this is true (if not the records will wrap around and become non-sense). Q-GIS is useful for inspecting shapefile attribute tables to discover issues and verify that your new shapefile works in an independent reader.

In my case data wasn’t available for all countries, so I padded with a default value (appended to the end of all records when adding the field) and then looped through and put the correct data in the records for which data was available.

Example here adding a new field for Numeric data and default data to all records. All my data is non-negative, so magic number “1” is for the decimal point.

    def addField(name, widthMinusPrecision, precision = 3, default = 0): 
        sf.field(name, 'N', widthMinusPrecision+precision+1, precision)
        # add default data all the way down to keep the shape file "in shape"
        for r in sf.records:

4th pyshp 1.1.4 workaround: delete() method doesn’t work, don’t use it.

Each shape is described by two pieces of data, linked together based on their index. When deleting a shape, both the record (with the meta data) and the shape (with coordinates etc) must be removed. If only one is deleted pyshp will add a dummy entry at the end and many of your records and shapes won’t line up anymore. To delete a shape, you must delete both the shape and the corresponding record. The delete method doesn’t do this, don’t use it, do it yourself:

    def deleteShape(n):
        del sf._shapes[n]
        del sf.records[n]

5th pyshp 1.1.4 workaround: Handle non-ascii character encodings yourself

pyshp doesn’t declare a character encoding when reading files, so they default to “ascii”. If you are using the Natural Earth shapefiles they have non-ascii characters and are encoded in Windows-1252. (See previous post for more info about the Natural Earth encoding.) I worked around this by looping over the records and encoding all strings to unicode:

    for r in sf.records:
        for i,f in enumerate(r):
            if isinstance(f, str):
                r[i] = unicode(f,'cp1252')

And then reversed this before saving the file via:

    for r in sf.records:
        for i,f in enumerate(r):
            if isinstance(f, unicode):
                r[i] = f.encode('cp1252')

6th pyshp 1.1.4 workaround: When looking at sf.fields adjust the index by one to ignore ‘DeletionFlag’

pyshp adds an imaginary field for internal state tracking to the beginning of the fields list. If you are looking up field names in this list to find indexes, you should correct your indexes accordingly, there is not actually a field called ‘DeletionFlag’.


After working around these bugs and massaging my country names to map from .xls to the names in the shapefile (17 cases of “Bolivia (Plurinational State Of)” == “Bolivia”) , I was able to use pyshp to generate a new shapefile with my data in it! Next up, cartogram-orama.

Unicode with HTML and javascript

OMG, really, character encoding problems again??! The adventure continues now in the browser.

First problem: I tried to use the d3 projection.js module, but including it gives me the error “Uncaught SyntaxError: Unexpected token =” in projection.js line 3. Looking at the file I am initially confused:

(function() {
  var ε = 1e-6,
      π = Math.PI,
      sqrtπ = Math.sqrt(π);

Until I noticed this module does a lot of fancy math with characters like ζ, μ, λ, π and φ. Ah ha! Perhaps this is my problem. Lo and behold, my lazy html didn’t declare a character encoding. The error was resolved by adding the following in the head of index.html:

<meta http-equiv=”Content-Type” content=”text/html; charset=UTF-8” />

Fast forward sometime later… Oh no! My old friend Cote d’Ivoire isn’t looking right:

Looks like my shapefile data, now converted to GeoJSON is still encoded in a non UTF-8 encoding. Switching the html encoding to <meta http-equiv=”Content-Type” content=”text/html; charset=ISO-8859-1” /> results in the tooltip rendering as expected, but clearly I am not willing to give up on using 3rd party code that is encoded in UTF-8 am I?

Fortunately there is another easy fix. Simply specify the encoding on the data file when I import it:

<script type = “text/javascript” charset=”ISO-8859-1″ src=”non-UTF-8_data_file.js”></script>

And all is well again:

Whew! Previous battles with character encodings came in handy.

Cartogram Basics

I am working on a Cartogram of the World with my friend Kim Ducharme. We are looking for something dramatic to show disparity between affluent countries and developing countries and a cartogram seems a great way to hit the message home. This is my first foray in to the world of cartograms, so here is some useful background.

A cartogram is a map where the areas of regions have been adjusted to represent some other metric of interest. They are intentionally distorted “maps” and yes there is controversy over them :).

Cartograms come in a few different flavors (see indiemaps summary):

  1. Non-contiguous Cartograms: Each object (state/country/etc) grows or shrinks independently of its neighbors. With the result being perfectly accurate, but the original map is filled with white space. Excellent history of Non-contiguous cartograms here.
  2. Dolring Cartograms: Replace the regions with circles or squares I won’t discuss these more.
  3. Contiguous Cartograms: Attempts to keep boundaries connected and distorts the shapes (often grossly) attempting to scale the country areas according to some metric. The canonical approach seems to be the Gastner/Newman diffusion method.
Our preliminary investigation showed that the Non-contiguous Cartogram was not a very satisfying image. Too much white space and just doesn’t have the punch of a wildly distorted contiguous cartogram.
Here are is an example of the diffusion method image generated by Mark Newman, this one represents Total spending on Healthcare, check out Mark Newman’s site for more in this family:

Having decided to pursue generating a contiguous area cartogram, first step was to find out how.

Selecting a Cartogram program:

There are a few options I found for generating a contiguous area cartogram:

  1. Download the source for Gastner/Newman’s cart program, compile and run it. Looks doable, but would need to massage my data into some grid format. Maybe someone else has made it easier…
  2. Apparently ArcGIS has a Cartogram Geoprocessing Tool based on the Gastner/Newman method too. But I don’t have a budget for fancy commercial software.
  3. ScapeToad: At last, a ready to go cartogram generating program also based on the Gastner/Newman method. This one has a GUI and is released under the GPL. Perfect!

Using ScapeToad to generate a cartogram:

Using ScapeToad is easy! It has simple instructions and I only ran it a few issues easy to work around. It uses ESRI shapefiles and will output the updated image as an ESRI shapefile (with error  data added in). I also found the ScapeToad documentation pretty helpful. The only annoying issue  on my system was the recently used files selection silently didn’t work. So when opening a shapefile (via “Add Layer…”) I had to always browse to the correct location. I will discuss about the “Mass” and “Density” options in another post. Otherwise there is not much to say, I will let ScapeToad speak for itself.

ScapeToad requires the shape file to have “perfect contiguity”, so find a suitable shapefile and test it before moving on. As discussed in a previous post, the Natural Earth shapefiles are now my go to. These files conveniently have some population data you can use to test the ScapeToad is working.

More on the real challenges, adding your own data to a shapefile, coming up…

Reprojecting maps with QGIS

I figured out how to reproject maps with Q-GIS and would like to celebrate with this lovely image, a US Atlas Equal Area Projection, which I thought was the most fun of the built in projections in Q-GIS 1.8.0:

If the Q-GIS documentation site is working it isn’t too hard to figure out how to reproject. Quick summary:

  • File >> Project Properties: You can set the project Coordinate Reference System (CRS), if you “Enable ‘on the fly’ CRS transformation” you can play around with the different projection options.
  • To set the CRS on a shapefile layer, right-click on the layer name and select “Set Layer CRS”, this is particularly useful if you don’t have ‘on the fly’ enabled, then your layers will not display if they are in a different CRS than the current project one, in which case you can change them here so they will display.

Note for practical equal area projections I found the Mollweide options seemed to work well, but don’t understand yet the difference between world and sphere options. Anyone?

Funny, it wasn’t untill I started looking at these equal area projections that I realized how big Russia is!

My introduction to unicode, and Natural Earth ESRI shapefiles

Working on a project for a friend I am adding data for different countries into a shapefile of the world. On a tip from Andy Woodruff, a cartographer I met at the Hubway Challenge Awards Ceremony, I switched to using the Natural Earth maps which are completely in the public domain. Great maps, with the added bonus of country names with non-ascii characters!

The data I started with is in Excel, with ALL CAPS country names. I needed to create a mapping from the .xls names to the names in the existing shapefile and then add new fields for my data and add it to the corresponding country shapes. Turns out in the excel file CÔTE D’IVOIRE is the only one with any fancy characters. Note, I am new to unicode, so I hope that renders as a capitol O with a circumflex in your browser too. The python csv module correctly reads the excel file as utf-8 encoded and so in python this name is represented with the string cote_str = u”C\xd4TE D’IVOIRE”. The ‘u’ prefix indicates it is a unicode string. When printed to my console using “print cote_str” it is rendered using the default encoding of my terminal of utf-8 and displays as desired: CÔTE D’IVOIRE. However, using the repr() method I can get at the details: (u”C\xd4TE D’IVOIRE”) and see that the unicode code point value for this character is 0xd4. However, if I encode the string into a utf-8 byte string, I can see the utf-8 encoding for this character (c394) as it would be stored in a utf-8 encoded file, see this unicode/utf-8 table for reference:

>>> cote_str.encode(‘utf-8′)
“C\xc3\x94TE D’IVOIRE”

Had I thought to look I would have seen it clearly documented that “Natural Earth Vector comes in ESRI shapefile format, the de facto standard for vector geodata. Character encoding is Windows-1252.” Windows-1252 is a superset of ISO-8859-1 (a.k.a. “latin1″). However, it didn’t occur to me to check and I ran into some unexpected problems, since several country names in this shapefile had non-ascii characters.

The pyshp module doesn’t specify an encoding and so the default for python is used which is “ascii”. So for example I end up with byte strings with non-ascii characters: “C\xf4te d’Ivoire”. When printed to the terminal it is rendered to utf-8, but since 0xf4 is not a valid utf-8 encoding it renders as: “C�te d’Ivoire”. More problematic other operations won’t work, for instance I need to compare this country name to the ones in the .xls file. Note I found it confusing at first that both unicode and latin-1 share encodings for values 0-255, but utf-8 has different encodings above 128 (because of how utf-8 uses variable numbers of bytes, the upper part of latin1 is not valid utf-8 at all, wikipedia’s description chart shows it well).

The raw byte string:

raw_c = “C\xf4te d’Ivoire”

can be converted to unicode with the proper encoding:

u_c = unicode(raw_c, ‘cp1252′)

which is now a unicode string (u”C\xf4te d’Ivoire”) and will print correctly to the console (because print is converting it to the correct encoding for the console).

Just playing about some more.

raw_utf8 = u_c.encode(‘utf-8′)

raw_utf8 now stores “C\xc3\xb4te d’Ivoire”, note that utf-8 needs two bytes to store the correct o. This will print looking correctly to my linux console because utf-8 is being used by the console.

However, in windows again I get something weird looking, because the windows command line is using code page 437 as the console encoding. Using u_c.encode(‘cp437′) gives me a binary string that prints correctly in this case “C\x93te d’Ivoire”. Having fun yet?

Moral of the story, debugging unicode can be confusing at first. Using unicode strings is clearer.

Tired of typing in ‘\xf4′ etc? You can change the default python from using ascii to using other encodings by adding a special comment in the first or second line of the file;

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# This encoding allows you to use unicode in your source code
st = u’São Tomé and Principe’

Here is a good reference on typing unicode characters in emacs.

Now I am less confused and have all the tools I need to work with these files shapefiles in the imperfect but still pretty functional pyshp module.

  1. Convert latin1 binary strings to unicode using unicode(s, ‘latin1′)
  2. Add the needed custom mapping entries by typing in unicode.
  3. Convert the unicode strings back to latin1 before saving the shapefile.

Hacky, but it works.

A better simple map

I am learning to use (a pretty buggy but functional python module for reading and writing ESRI shapefiles) and Quantum GIS. As a quick demonstration I replotted the data from an earlier map. Q-GIS makes it pretty easy to adjust the appearance. The world map shapefile is from Natural Earth.

This time I generated a shapefile from python directly, super easy (I will highlight problems with pyshp in a future post, but creating this simple file worked fine, although doesn’t seem to define a CRS, I am pretty sure it is WGS 84). Here is the code:

import shapefile
w = shapefile.Writer(shapefile.POINT)
max_len = max([len( for s in stations])
for s in stations:

Picture viewing web page

I wanted to share pictures taken from our Antony and Cleopatra Reading this week and decided to create a page with thumbnails and links to the full pictures. Sure there are programs out there that do it and host for you, but it is so hard to get at the full res images and I guess they just annoy me, so I wanted to do it old school. Also I figure it is a good opportunity to practice some of the skills I have been exploring lately.

1) Write a shell script to shrink the image files and create thumbnails. Well actually putting it in a batch file is overkill, but it’s the first shell script I have created, pretty simple:

  • create a file with .sh extension and give it executable permissions with “chmod -x [name].sh”
  • add “#!/bin/sh” at the top
  • add commands that you want to execute from the commandline below

2) shrink all the files, I used imagemagick (convert) to do this the -thumbnail option is designed for creating thumbnails and the x300 makes the final image 300 pixels tall and maintains the aspect ratio. The command I ended up using was:

ls *.JPG | xargs -0 | xargs -I {} -n1 convert {} -thumbnail x300 thumbs/thumb_{}

3) I used a little java script to generate the web page.

  • first created a list of all the files using underscore.js, then
  • used d3.js to populate the appropriate number of links to display the thumbnails as links to the full size images

The resulting picture page is not fancy, but functional.