Featured

Fuel shortages, Google Maps and adapting your tools

Archaeologists have always used tools invented for other purposes. Our ‘signature’ tool, the trowel, was designed for brick-laying; our hoes, mattocks, shovels and wheelbarrows are gardening tools; and I doubt the inventors of the JCB dreamt that with the right bucket and a skilled operator their heavy earth-moving equipment could delicately strip off a few tens-of-centimetres of topsoil to cleanly expose archaeology in the subsoil beneath. What is true of physical tools is also true of technological ones; total stations and computer-aided design were created for construction; satellite imagery had its genesis in espionage; GIS were first used by geographers. Even our theoretical and methodological foundations are often borrowed from others. As archaeologists we specialise in taking something made for something else, and tweaking it to answer our questions. Adapting an existing tool to answer our research questions is as archaeological as brushing dirt from a precious artefact and, as I recently discovered, is widely applicable to a variety of situations beyond the excavation, lab or library.

Seeking petrol

At the time of writing, the UK is currently experiencing a temporary fuel shortage, resulting from a combination of panic-buying and a shortage in HGV drivers. I live in a suburban area, but public transport is limited and I have a pre-schooler, so I have always done a modest amount of driving. The fuel shortage caught us unawares with limited petrol in the car and after a few days of taking my little one to school we were running seriously short. At this time fuel deliveries were still being made, but the number of people seeking fuel and the intermittent nature of the deliveries meant that it was impossible to know which petrol station would have fuel. I needed to find a way to determine exactly which petrol station might have what I needed before I left. Where could I find real-time information on which petrol stations had fuel? How could I locate the closest one?

Real-time feedback in Google maps

Thanks to years of archaeological research, I have a long history of looking at maps and satellite imagery and I’m used to combining data, maps and visual imagery to address research questions. So when I was faced with a very real problem my brain automatically thought along the same lines. Since I was looking for the closest petrol station with fuel, the problem was obviously a question of mapping. Google maps is one of the most powerful mapping tools available for simple tasks and offers real-time information, it was a sensible place to start looking for a solution.

Google maps image showing the Southend-on-sea area, with petrol stations marked.
Google Maps showing petrol stations near my home in Leigh-on-sea. Screenshot taken at 12.30pm on Monday 4 October (Google Map data ©2021).

When teaching basic GIS I often use Google maps as an example of a deceptively simple but immensely powerful GIS. Behind the Google maps interface is an incredibly complex programme collating vast quantities of data, from satellite imagery and road maps, to data about speed limits and real-time information about traffic movements. Despite this complexity and vast processing power, it appears simple to us because the interface places substantial limits on what we can do with it; see a map of the area around us, search for a location, navigate from one place to another, or find a simple amenity, like a petrol station. Google Maps is immensely powerful but setup to be used in certain specific ways. It contains the locations of nearby petrol stations (as in the image above) and real-time traffic information (image below), but its not setup to find petrol stations with fuel. If I wanted do that I would need use it in a way its designers had never intended. I would need to adapt this tool to access the data I needed, just like generations of archaeologists before.

Google maps image showing the Southend-on-sea area, with traffic information turned on.
The borough of Southend-on-sea in Google Maps, with traffic information turned on. Screenshot taken at 12.45pm on Monday 4 October (Google Map data ©2021)

Adapting the tools

Adapting the data from Google Maps to answer my question necessitated current background knowledge of the situation and its impact on my society. The one thing everyone in any urban or suburban area knows about the petrol shortage is that any petrol station with fuel rapidly creates a jam along the adjacent road as people queue up to get fuel and block the road. So I combined the petrol station locations with the traffic information to identify petrol stations with queues outside. Since people don’t queue outside a petrol station without fuel, those with queues must have fuel. In the image to the right Tesco Petrol Station and the West Street BP to the right of the map just above the ‘Southend’ label, both have queues outside. The Shell to the right of Tesco does not. We promptly went to the West Street BP and fuelled-up without difficulty.

Google maps image showing a detail of the Southend-on-sea area, with petrol stations marked and traffic information turned on. Some petrol stations show a red queue of traffic approaching their entrances, while others do not.
Google Maps extract from 28 September 2021, 8pm. (Google Map data ©2021)

‘Improper use may cause damage!’

As when using any tool in a way its creators did not intend, a certain amount of local knowledge, experience and common sense is required. In the image to the right the BP with a Wild Bean Cafe in the bottom left also has queues shown to the west of it, but I can’t be sure those indicate it has petrol. It’s on the A13 and close to a junction. The A13 is a major road that is often slow and can have queues and jams almost anywhere on it for any reason, and the traffic lights often produce short local jams like those shown in the image. So if I was really desperate to go straight to a petrol station with fuel, seek out queues at petrol stations on a minor roads some distance from intersections. The BP petrol station on West Street, Prittlewell, is an excellent example. The West Street BP is centre right, just above the ‘Southend’ label in the image above right. When I checked traffic information against petrol stations I found a dark red tail leading away from the West Street BP (centre right in the image below). West Street is not a major road and the BP is not close to a major intersection. The only likely reason there’s a queue leading straight to that petrol station at 8pm is that those people are waiting for fuel.

Checks and balances

Google maps image showing a detail of the Southend-on-sea area, with petrol stations marked and traffic information turned on. Some petrol stations show a red queue of traffic approaching their entrances, while others do not.
Google Maps of the area near Leigh-on-sea showing petrol station locations and traffic information. Screenshot taken at 1pm on Monday 4 October (Google Map data ©2021)

Its possible to double-check that a petrol station has fuel using the powerful statistics held in Google maps. At the time of writing it is 1pm on a sunny weekday. There should not be a lot of unusual traffic or unexpected jams that are unrelated to fuel queues and might confuse me, but if it was rush-hour I’d have to be more careful. Checking the petrol situation at the present (image above) it seems that the Tesco Petrol Station and Shell in the centre right of the image have fuel, while the Esso MFG Kent Elms does not.

Details of the Tesco Petrol Station in Southend on sea showing its address, opening hours, phone number and times when it is busiest.
Tesco Petrol Station statistics from Google Maps on 4 October 2021 at 1.25pm.
Details of the Tesco Petrol Station in Southend on sea showing its address, opening hours, phone number and times when it is busiest.
West Street BP petrol station statistics from Google Maps on 4 October 2021 at 1.25pm..

I can confirm that the traffic information reflects queues for fuel rather than any other traffic incident by checking the real-time statistics in Google Maps (this was Paul Barrett’s idea). If you click on the little marker balloon for a likely petrol station and scroll down, there’s a little graph showing popular times. If the queue on the traffic information is for fuel, the graph should show that the petrol station is currently ‘busier than usual’. If we check on the Tesco Petrol Station for now 1pm on 4 October 2021, we find it is ‘busier than usual’ (image above left), confirming the evidence of the traffic information, that showed a queue outside it. This contrasts quite nicely with the BP on West Street, which is now running out or is out of fuel and is therefore ‘less busy than usual’ (image above right).

Adapting and mis-using tools

Aside from being an interesting use of an incredibly powerful free GIS, why is my solution to a personal fuel crisis of archaeological or Egyptological significance? I believe there are two reasons. The real-world value of humanities and social science disciplines like archaeology and Egyptology are often questioned and those who study them expected to explain how their supposedly arcane subject prepares them for the real world of work. I developed this particular method of solving my real-world fuel shortage problem because of my long history of looking at maps and satellite imagery and combining various sources of data and visual imagery to address research questions. While students learn many valuable skills studying archaeology or Egyptology, the ability to marshal varied sources of data in textual, statistical and visual formats; and judiciously adapt tools for new purposes, is highly valuable in many professions.

Secondly, my method of finding petrol stations with fuel follows the same model of adaptation that is common in archaeology and Egyptology. I took an existing, common tool (Google Maps) and combined two established features of that tool (real-time traffic information and the local search function) with specific cultural knowledge (that petrol stations with fuel attract queues) to extract information on which petrol stations were likely to have fuel. I improved the rigour of my method with additional background cultural and cartographic knowledge (of the main roads and traffic hotspots) and checked it against an independent dataset (Google Maps real-time statistics on how busy locations are). The result is an efficient and effective method of obtaining information I did not previously have direct access to but which was present in existing datasets, all without developing new software or diagnostic tools. For me this is what GIS, landscape and archaeological research is all about!

Featured

Finishing the map, georeferencing the pyramid of Djedefre

In a previous post I described how I georeferenced a difficult map of Abu Rawash. During that process I had to ignore the pyramid of Djedefre because it was drawn at a different scale to the rest of the map. Here and in this video I discuss how I subsequently georeferenced the pyramid of Djedefre at the correct scale.

The map of the pyramid of Djedefre was cropped from map I in Porter and Moss’ (1932) Topographical Bibliography volume IIIi, which is featured in my previous post.

Comparison of the angle of the causeway after accurately georeferencing the pyramid (left) and the angle of the causeway in the satellite imagery (right). Map cut from Porter and Moss 1932, volumne IIIi, map I.

First, I identified the archaeological features in the map and the satellite image. Although the pyramid is very clear in the satellite imagery, this process was complicated by the long ramp or causeway heading north from the pyramid. Initially I assumed this was the pyramid’s causeway to the Valley Temple, but the angle of this feature in the map, and the angle in the satellite imagery were different. This made me wonder if the feature I could see in the satellite imagery was some kind of recent Decauville track for removing spoil from the pyramid, and the actual causeway ran at a different angle and had been removed. Having georeferenced the map I think the feature is probably the pyramid causeway in both cases, the angle in the map being incorrect due to the differences in scale between the drawing of the pyramid and the rest of the map. If you review the image of the georeferenced map in my previous post you will see that the end of the causeway as drawn in the map lines up with the end of the feature in the satellite image, even though the pyramid is incorrectly drawn. This suggests that the angle of the causeway became misaligned when the pyramid and the rest of the map were joined together. Nevertheless I chose to ignore the causeway when georeferencing the pyramid, because its questionable accuracy would make georeferencing more difficult and it was readily visible in the satellite image anyway.

I then scaled the image to the correct scale to align it to the satellite imagery (from 1.20 minutes in the video). As I note at 4.59 in the video, one thing to be aware of when georeferencing maps is that the lines of the map occupy space within the GIS – so the line of the enclosure wall of the pyramid complex represents 3m on the ground after georeferencing. This can make it difficult to align the map with satellite imagery, particularly if the map only covers a small area and/or is cropped from a much larger map.

Once I determined approximately the correct scale (1:2250) I began linking the ground control points in the map and satellite imagery (from 4.15 minutes in the video). This revealed further inaccuracies and forced me to make decisions about which points in the map I believed were more accurate than others. In this previous post I discussed the importance and limitations of RMSE. The georeferencing of the map of the pyramid of Djedefre really emphasises how RMSE and residuals can be used to improve georeferencing, and also the limitations of the process. I used the RMSE and residuals, combined with the visual position of the map on the satellite image, to test the ground control points (from 10.08 in the video). They rapidly revealed that parts of the pyramid complex had been drawn inaccurately in relation to each other. After noting that the satellite pyramid and south-west corner of the enclosure were in dashed lines, I opted to set the ground control points elsewhere as it seemed likely that the satellite pyramid was more speculatively drawn. Testing various ground control points also revealed that the enclosure wall around the complex was drawn closer to the pyramid than it really is, forcing me to chose whether to include the complex enclosure wall in the ground control points or concentrate on the pyramid. I chose to focus upon the pyramid and mortuary temple, and rectified the map with an RMSE of 3.59, which was an improvement on a previous attempt, but still far from the 0.75m RMSE which would represent the 1:3000 ideal (Conolly and Lake 2006, 82-83). The inaccuracy in the map, its scale and the resolution of the satellite imagery are all contributors to this high RMSE. Depending on what I need to do with the map, I may seek out a more recent map or re-georeference it. Georeferencing a map this small, with this many inaccuracies, to satellite imagery, is always going to be difficult and likely to produce a high RMSE.

A satellite image of Djedefre's pyramid complex overlaid with the plan from Porter and Moss 1932, map I.

Acknowledgements and References

Conolly, J. and Lake, M. 2006. Geographical Information Systems in Archaeology. Cambridge.

Porter, B, and Moss, R. 1932, Topographical Bibliography of Ancient Egyptian Hieroglyphics, Texts, Reliefs and Paintings III: Memphis 1. Abu Rawash to Abusir. Oxford.

Maps and images throughout this blog post were created using ArcGIS® software by Esri. ArcGIS® and ArcMap™ are the intellectual property of Esri and are used herein under license. Copyright © Esri. All rights reserved. For more information about Esri® software, please visit http://www.esri.com.

All the satellite imagery used is ArcGIS World Imagery. Sources: Esri, DigitalGlobe, GeoEye, i-cubed, USDA FSA, USGS, AEX, Getmapping, Aerogrid, IGN, IGP, swisstopo, and the GIS User Community.

Featured

Errors, inaccuracies, resolution and RMSE: Georeferencing a difficult map of Abu Rawash’s pyramid and cemeteries

In a previous post I introduced the georeferencing work I was doing, with a video of me georeferencing Porter and Moss’ map of the Cemetery F mastaba field at Abu Rawash. Here I delve into the process a little more, with help from a further video, which shows me georeferencing a more difficult map of Abu Rawash using ArcGIS basemap World Imagery.

The map I am georeferencing in those videos is map I in Porter and Moss’ (1932) Topographical Bibliography volume IIIi. It shows the entire area of Abu Rawash from the pyramid of Dejedefre, to the north-west cemetery in Wadi Qaren and the village of Abu Rawash on the edge of the cultivation.

Map of Abu Rawash showing the pyramid to the south-west, and various cemeteries in the desert around.
Map I from Porter and Moss’ 1932, Volume IIIi, of the Abu Rawash area.

Accuracy and precision.

When working through the georeferencing process, its important to carefully consider the precision and accuracy you are aiming for, taking into account likely distortions in the image to be georeferenced; the resolution and accuracy of the data (the satellite imagery) you are georeferencing with; and the ultimate function of the georeferenced image.

All data, even GCP collected on site with differential GPS, have some level of error in them. What matters is that they are sufficiently accurate and precise for the task you have in mind. Accuracy and precision are also different. Accuracy refers to whether something is correct. In other words, is the map you are georeferencing in the right place? Precision is easiest to think of as resolution or the level of detail. Something can be very precise but very inaccurate, or something can be very accurate and very imprecise. It is accurate to say I live in the United Kingdom, but it is not very precise. It is precise to say I live in N7 6RY (A postcode in Finsbury Park, London) but that is not accurate – I do not live there any more.

The accuracy of the georeferenced image can be affected by drafting or composing errors in the original map, or by distortions introduced during printing or digitising. Such inaccuracies and distortions affect how well the map can be overlaid on the satellite imagery. The satellite imagery can also contain distortions. Gross and obvious distortions, like the kinks in the Deir el-Bahri temples I published in a previous post and distortions due to the angle of the satellite to the ground affect the georeferencing process.

As with accuracy, the precision of the georeferenced image is affected by the precision of the map we are georeferencing, and the resolution of the satellite imagery. If the map is sketchily drawn or details are missing this may make it more difficult to locate precisely. The resolution of the satellite imagery (or any other ground control data) also affects the precision of the georeferenced image. Georeferencing requires that we line up the map with the same features in the satellite image. The more precisely a feature appears in the satellite image, the more precisely we can locate the map we are georeferencing. The pixel size (resolution) of the satellite imagery therefore places considerable limits upon the precision of the georeferenced image.

Locating the archaeological features at Abu Rawash

It feels like it ought to be easy to identify the pyramid of Abu Rawash and associated cemeteries. After all, pyramids are not known for their discretion, particularly once they’ve been excavated. While the pyramid of Abu Rawash is pretty clear in the satellite imagery, the huge amount of quarrying and agricultural and housing development in the Abu Rawash area made it difficult to identify the geographical and archaeological features in Porter and Moss’ 1932 map. The Survey of Egypt 1:25,000 scale map of 1942 shows how little development had taken place 80 years ago.

Map of the Abu Rawash area from 1942 showing minimal modern development in the desert.
Abu Rawash in 1942 (Survey of Egypt 1:25,000 scale map of Kirdasa).

In contrast the modern satellite image shows the pyramid and cemeteries as islands of archaeological landscape in a highly developed area.

Image of the Abu Rawash area showing considerable quarrying and development around the pyramid and cemeteries.
The area of Abu Rawash today, note the considerable development and quarrying in the area.

This made it more difficult to relate Porter and Moss’ map to the satellite image, although the preservation of the pyramid and the cemeteries did help (see from 0.35 minutes in the video).

Scaling the map for georeferencing

Once we identify the area, we need to scale the map to fit that area in the satellite imagery. You can see me undertaking this task from 1.10 minutes in the video. During the process, I discovered an inaccuracy in Porter and Moss’ map – the pyramid of Djedefre had clearly been drawn at a different scale to the rest of the image. Drafting and composing inaccuracies of this type are frustrating but do occur with historic imagery. Other sources of inaccuracies include distortions introduced during the scaling of maps for publication and when publications are scanned or photographed to generate digital images. Photography is particularly problematic as the camera lens needs to be parallel to the image to avoid distortion, but even scanning can produce minor inaccuracies. Dealing with these inaccuracies often means adjusting the georeferencing, splitting an image or ignoring part of the map during the georeferencing process. In this case I chose to georeference the map, while ignoring the pyramid and subsequently cropped out the pyramid and georeferenced it separately.

Adding control points (GCP)

Once Porter and Moss’ map had been scaled to approximately the right scale, it was aligned more precisely to the satellite imagery using ground control points (GCP). These appear in ArcGIS as ‘Links’, and operate essentially as pins. You select a point in the map and then select the same point in the satellite image and ‘pin’ them together. You can see me undertake this process from 7.20 minutes in the video. Ideally, it would be possible to locate these links very precisely in each set of data, but in this example, we are constrained by the contents of Porter and Moss’ map, which does not include many clear points that can be related precisely to points in the satellite imagery. The resolution of the satellite imagery is also a factor. The ArcGIS Basemap World Imagery uses a variety of satellite imagery sources, but the highest resolution of any commercial satellite imagery is currently c. 30cm and much of the imagery is likely to have a resolution of c. 40-50cm or more. This means that the pixels of the satellite image represent 40-50cm on the ground. Any feature smaller than that is invisible, and features that are only slightly larger are difficult to identify. Another effect of the satellite imagery resolution is that when we zoom in close the satellite imagery appears blurry, and a point becomes more difficult to locate than when zoomed out (you can see the effect of this from 8.45 in the video).

Root Mean Square Error (RMSE)

Once we have added four links in ArcGIS, we can open the link table and see a RMSE for the entire map in the top box and the residuals for each point in the right column of the table. Turning off links or adding new links will alter the position of the map and the RMSE accordingly (from 10.30 in the video). The RMSE represents ArcGIS’ calculation of the fit between the actual and desired link positions (Conolly and Lake 2006, 82-83). In simple terms ArcGIS uses the first three links to estimate where it thinks any further links should be. It then calculates the residual for each point as the difference between where you placed a link and where the map ended up based on the other links that have already been placed. The RMSE is the product of all the residuals. Although RMSE is useful, it’s important to recognise that it is reliant upon the accuracy and precision of the map and the satellite imagery. If there are inaccuracies in either, they will increase the RMSE. It is also reliant upon the locations and positioning of the points you choose. The old adage of ‘junk in, junk out’ definitely applies and it is entirely possible to have a low RMSE and a very inaccurate and imprecisely positioned map. So while you can reduce your RMSE by removing links with high residuals and adding new links, it is sometimes better to accept a higher RMSE and keep an important link, recognising that the higher RMSE is due to inaccuracies in the map. Alternatively, it may be necessary to chose which ground control points you believe are more accurate and only link to them.

We aim for an error of less than 1:3000 so for an original image at a scale of 1:15000 an RMSE of under 5 (i.e. 15000/3000) is ideal (Conolly and Lake 2006, 82-83). Ideally we would use the scale given in the original image, but Porter and Moss do not include scale information so we have to work with the scale we established during georeferencing. When I scaled this map I settled on a scale of 1:9000, so any RMSE under 3m would be very acceptable. Here our RMSE is slightly above 3m, which is not unreasonable given the inaccuracies in the map and the difficulty of locating very precise control points due to the resolution of the satellite imagery and changes to the landscape. I subsequently repeated the georeferencing and obtained an RMSE of 2.88, but reducing the RMSE by a large amount is not always possible depending on the scale and accuracy of the map, and the resolution of the satellite imagery. The map of Cemetery F, for example was at a scale of just over 1:500, meaning its RMSE should be 0.16m or under, but I was only able to get it to 0.3m. Nevertheless, under the circumstances that is acceptable because of the resolution of the satellite imagery, which makes it impossible to place a point more precisely than within 0.3m. This is compounded by the imprecise edges of certain archaeological features in the satellite imagery, such as the mastabas of Cemetery F or the satellite pyramid of Djedefre, and any inaccuracies or distortions in the maps. In such cases it is important to be aware of known inaccuracies and distortions in the map and satellite imagery or you can be driven to distraction trying to get inaccurately positioned features to line up.

Ideally, if the RMSE is too high and cannot be reduced, we would seek an alternative source of data, but such data does not exist for some of these sites. In those cases it is much better to have a slightly less than ideally georeferenced map, than none at all. It is also important to be aware of the purpose of your georeferenced map. In this case the relatively modest aim was to locate archaeological features to within 10m, which is achievable with the accuracy of the maps and the resolution of the satellite imagery.

Overall I was satisfied with the georeferencing of the Abu Rawash map. It was a very difficult map to georeference; hard to locate due to the changes to the landscape; difficult to scale due to the inaccuracy in the pyramid; and difficult to find enough precise features to use as GCP links . Nevertheless, the final georeferenced version gives useful insight into the archaeological landscape. With careful thought and reference to the underlying satellite image, it will be possible to locate any relevant archaeological features during the rest of the project.

A map of the Abu Rawash area, overlying a satellite image. The pyramid is clearly at the wrong scale and angle compared to the rest of the image.
Final georeferenced version of Porter and Moss’ 1932 Volume, IIIi, map I of Abu Rawash.

Acknowledgements and References

Conolly, J. and Lake, M. 2006. Geographical Information Systems in Archaeology. Cambridge.

Porter, B, and Moss, R. 1932, Topographical Bibliography of Ancient Egyptian Hieroglyphics, Texts, Reliefs and Paintings III: Memphis 1. Abu Rawash to Abusir. Oxford.

Maps and images throughout this blog post were created using ArcGIS® software by Esri. ArcGIS® and ArcMap™ are the intellectual property of Esri and are used herein under license. Copyright © Esri. All rights reserved. For more information about Esri® software, please visit http://www.esri.com.

All the satellite imagery used is ArcGIS World Imagery. Sources: Esri, DigitalGlobe, GeoEye, i-cubed, USDA FSA, USGS, AEX, Getmapping, Aerogrid, IGN, IGP, swisstopo, and the GIS User Community.

Featured

Wonky Giza pyramids: Oblique satellite imagery and georeferencing

I’m currently working on a project georeferencing (or georectifying) a lot of historic maps published in Porter and Moss’ Topographical Bibliography. I’m georeferencing these maps with ArcGIS basemap World Imagery and as a result have spent many days looking at satellite images of Egypt.

Satellite imagery is a hugely valuable resource, but it can be misleadingly precise. One feature of satellite imagery that isn’t immediately obvious is the problem of parallax. Parallax is the displacement of an object when seen from different positions. It’s incredibly useful in astronomy, but more of a problem in geodesy. Maps provide a vertical view of surface of the earth, flattened onto a flat plane below the imaginary godlike viewer. Satellites (and aeroplanes) fly across the curving surface of the globe taking images as they move. This means that some or all of the each satellite image is taken from an oblique angle and that can produce parallax.

The parallax is really clear in a georeferenced map of the Giza pyramids. In the satellite image below the points of the three Giza pyramids are to the north-west of the points in the overlaying georeferenced map. This is because the satellite was at a slightly oblique angle to the ground of the Giza plateau when the image was taken. As a result, when I georeferenced this map I had to be careful to line up the map with the corners of the pyramids to ensure the best accuracy. If I had used the tops of the pyramids my map would have been misaligned.

Map of the Giza pyramids overlying a satellite image of the area.
Georeferenced map of the pyramids of Giza, overlaid on the ArcGIS basemap satellite imagery. Note how the tops of the pyramids in the satellite image are offset to the north-west compared to the map. (Map III of Porter and Moss 1932, Volume IIIi)

Acknowledgements and References

Porter, B, and Moss, R. 1932, Topographical Bibliography of Ancient Egyptian Hieroglyphics, Texts, Reliefs and Paintings III: Memphis 1. Abu Rawash to Abusir. Oxford.

Maps and images throughout this blog post were created using ArcGIS® software by Esri. ArcGIS® and ArcMap™ are the intellectual property of Esri and are used herein under license. Copyright © Esri. All rights reserved. For more information about Esri® software, please visit http://www.esri.com.

All the satellite imagery used is ArcGIS World Imagery. Sources: Esri, DigitalGlobe, GeoEye, i-cubed, USDA FSA, USGS, AEX, Getmapping, Aerogrid, IGN, IGP, swisstopo, and the GIS User Community.

Featured

Shifting mastabas: Georeferencing a plan of a Fourth Dynasty Egyptian mastaba cemetery, at Abu Rawash.

I am currently working on a project to georeference (or georectify) maps of various Egyptian sites from Porter and Moss’ Topographical Bibliography (which can be found online at this Griffith Institute website). Georeferencing is something of a Cinderella job in geographic information systems (GIS) work – its important, but is often ignored in favour of more exciting methods and results. So for those who haven’t had the (sometimes dubious) pleasure of georeferencing a map for themselves, I’m making some videos of the process and uploading them to my own YouTube channel. The first video is available now and features me georeferencing a cemetery at Abu Rawash, north-west of Cairo.

Abu Rawash

Abu Rawash is the site of the pyramid of Fourth Dynasty Pharaoh Djedefre with cemeteries dating from the Early Dynastic period onwards. Cemetery F, like the pyramid, dates from the Fourth Dynasty and contains the high status mastaba tombs of a number of important royal courtiers. The outlines of these mastabas remain visible in the satellite imagery, with their burial shafts appearing as black marks in the centre of the structure.

Cemetery F, as it appears now in satellite imagery. The outlines of the rectangular mastaba tombs are clearly visible, most with two burial shafts in the centre.

Cemetery F was excavated by Bisson de la Rocque, and it is his plan that Porter and Moss include as Map II[1] of volume IIIi of the Topographical Bibliography:

Plan of Abu Rawash Cemetery F aligned and scaled to the mastaba field in the satellite image. (Published in Porter and Moss, 1932, MapII).

Georeferencing

Georeferencing is the process of taking an image and providing it with coordinates that allow the image to be correctly positioned in relation to other geographic data. Most of the historic sketches, excavation and survey plans made by generations of past archaeologists exist as published images. Georeferencing those images is often the first task in collating archaeological data and relating it to modern maps, survey data and satellite imagery.

My task was to use the GIS to locate Porter and Moss’ plan on the satellite image of the mastaba field, allowing, me to obtain geographic coordinates for any of the tombs within it. The georeferencing process I used divided into 3 parts: locating the archaeological features from the Porter and Moss map in the satellite imagery from 2:07 in the Cemetery F video); scaling the Porter and Moss map to the approximately the correct scale (from 2.55 in the Cemetery F video); and then using ground control points (GCP) to link locations on the Porter and Moss map to the same points in the satellite imagery (from 4.30 in the Cemetery F video). This task was complicated by the lack of scale in Porter and Moss’ (1932, Map II) published image (the scale in the image above has been added by me after georeferencing) and the resolution of the satellite imagery, which makes precise location of ground control points difficult at these scales. Nevertheless, the mastabas were relatively obvious in the satellite imagery and georeferencing was therefore easier than it might have been.

The video of me georeferencing mastaba Cemetery F at Abu Rawash, is now available on my YouTube channel and the next image shows the finished project, with the map from Porter and Moss overlaid on the satellite image from the ArcGIS basemap World Imagery layer.

Porter and Moss’ 1932 Map (II[1]) of Cemetery F at Abu Rawash, georeferenced and overlaid upon the mastabas as they appear today in the satellite imagery.

Acknowledgements and References

Porter, B, and Moss, R. 1932, Topographical Bibliography of Ancient Egyptian Hieroglyphics, Texts, Reliefs and Paintings III: Memphis 1. Abu Rawash to Abusir. Oxford.

Maps and images throughout this blog post were created using ArcGIS® software by Esri. ArcGIS® and ArcMap™ are the intellectual property of Esri and are used herein under license. Copyright © Esri. All rights reserved. For more information about Esri® software, please visit http://www.esri.com.

All the satellite imagery used is ArcGIS World Imagery. Sources: Esri, DigitalGlobe, GeoEye, i-cubed, USDA FSA, USGS, AEX, Getmapping, Aerogrid, IGN, IGP, swisstopo, and the GIS User Community.

Ditch that total station and grab your tablet? ArcGIS Collector and QGIS QField mobile-GIS survey apps reviewed.

Several GIS platforms now offer apps that synchronise GIS-based fieldwork between your computer and your phone or tablet. With options to add differential GPS equipment to your mobile GIS kit and the capacity to take your basemaps and other data with you into the field, mobile-GIS eliminates the disconnect between data capture and GIS. So how do the latest mobile-GIS apps measure up in the field? Here I review two that I tested recently during fieldwork at the classical site of Olynthos, during ongoing fieldwork in Greece.

Hannah_DGPS
Programming the differential GPS on the terrace of the Marsam Hotel, Luxor.

There have long been efforts to replace large and complex survey kit with smaller handheld devices. Until recently these typically involved either a substantial cost for the procurement of specialist hardware and software (such as the Leica Zeno and ESRI ArcPad) or experience in the management (and often programming) of open-source data solutions (such as many described in this discussion in GIS Stack Exchange). If you didn’t have the financial or programming resources to use these, then you were stuck with combining survey equipment, like total stations or differential GPS (see image left) with several different software programmes in order to create, process and import your data in a GIS (Geographic Information System). Once fieldwork data was incorporated into the GIS, it often required further processing to produce GIS-friendly layers for analysis and presentation and if the data was collected by a non-GIS specialist there were many opportunities for misunderstandings about the format required for GIS compatibility, resulting in stress and additional delay for one or both parties.

The advent of mobile phone and tablet compatible mobile-GIS apps has opened a range of new possibilities for recording fieldwork directly in a mobile-GIS platform, synced to your online or desktop GIS. During recent fieldwork at the site of Olynthos, Greece I trialed two different mobile-GIS apps ArcGIS Collector and QGIS QField on my Samsung Galaxy tab S2, running Android 7.

The fieldwork

The Olynthos Project is an interdisciplinary research project in northern Greece directed by Lisa Nevett of the University of Michigan, Zosia Archibald of the University of Liverpool and Bettina Tsigarida of the Ephorate of Pella. The project combines the excavation of houses on the north and south hills of the classical town, with field surface survey of the surrounding countryside and various scientific analyses (Nevett et al. Forthcoming). The project has typically used total stations for recording on site and printouts of Google Earth for identifying surveyed fields, with data from both sources incorporated into the project GIS in subsequent stages. Tablet-based mobile-GIS survey is not appropriate for precise recording of archaeological features in the excavation trenches, as GPS units integrated into tablet computers are only precise to c. 2.5-5m. But once combined with a satellite image basemap, this level of precision is perfectly acceptable for recording surveyed fields in the surrounding countryside or surface collection in grid squares on the unexcavated parts of the site.

The Olynthos Project began using ArcGIS Collector for surface survey following initial development of the Olynthos Project ArcGIS Online group and ArcGIS Collector maps by Peter Knoop and Caitlin Dickinson of the University of Michigan. The recent field season in July 2017 gave me the opportunity to test ArcGIS Collector against QGIS QField during archaeological fieldwork.

Mobile-GIS

ArcGIS Collector and QGIS QField are essentially simplified versions of their big brother desktop GISs, optimised for field data collection on a small mobile device with limited connectivity and processing power. Both apps operate on similar principles. You produce layers in your desktop GIS, convert them and upload them (directly or via the web) to your mobile device. The Collector and QField apps then run these ‘mobile’ versions of your GIS projects enabling you to a view, and create GIS layers in a format that is suitable for smaller screens and simpler for mobile devices to process. Both apps are designed for creating data in the field using the device’s GPS or a separate unit compatible with it. Both raster and vector data are supported and require processing for inclusion in the mobile app, raster data usually requires tiling to ensure it displays efficiently.

Despite their many similarities there are several significant differences between ArcGIS Collector and QGIS QField, not least of which is that Collector is proprietary and requires you (or your organisation) to have an online subscription, while QField is free and opensource (like all QGIS products).

ArcGIS Collector

Since many archaeologists, archaeological units and universities already have ArcGIS licenses and online subscriptions ArcGIS Collector is likely to be a significant player in future mobile-GIS for archaeological fieldwork. It requires you to have an ArcGIS Online account, through which the uploading and downloading or your mobile GIS data is managed. Setting up your Collector app is therefore a staged process, first you must share the layers you want to use to your ArcGIS Online account. Then create a map in your online account with a basemap (either ERSI ArcGIS imagery or your own tiled raster) and the layers you want to see or edit in the field. Finally you download that map to your Collector app on your mobile device. This process is a little cumbersome and confusing, particularly for those who haven’t used ArcGIS Online before. It’s easy to upload a layer but then fail to share or publish it correctly for you or others to view or edit it in Collector. This can be irritating, but there are online help pages and the usual forums (such as the ever-brilliant GIS stack exchange) and once you’ve created one map for Collector, familiarity makes the process a lot easier.

ArcGIS_Collector_interface
The ArcGIS Collector interface, showing the point data overlaid on ESRI ArcGIS imagery, with the attributes (including photo) on the right.

The good news is that once you’ve grasped the online part of the process, the interface (above) is clean and easy to use and using the app in the field is incredibly simple and straightforward. Once the app is linked to your ArcGIS Online account it identifies all the maps you and others in your organisation/group have created and offers you the opportunity to collect data using them.

You can operate Collector in two modes. If you just click on a map in Collector, it loads the map directly from your ArcGIS Online account via wifi or mobile data. This is straightforward, but if you don’t want to use mobile data or the connection is slow, then it’s a pain. Alternatively you can click ‘download’ on the map and download a section of it to Collector for offline editing. With a downloaded map you can use a basemap from your ArcGIS Online account, from ESRI ArcGIS imagery or from a raster on your device. You pick the geographical area you’re working in and the level of detail you need for your fieldwork and Collector will download just that area of your AcrGIS Online map so you can work offline, saving you time and mobile data. Once you finish recording in the field you can sync your downloaded map, uploading all your edits to their respective layers in ArcGIS Online (if you use Collector without downloading, the syncing occurs in real time via wifi or mobile data). You then return the edits to your computer by opening ArcGIS Online in your desktop GIS and adding your edited layers. The process within the app is generally slick and comfortable but a major issue with Collector is that it doesn’t currently support snapping. This necessitates various topological checks and editing of your data in ArcGIS desktop after you’ve downloaded it. I found this to be a serious deficiency in the ArcGIS Collector format, which is all the more surprising given that Collector is substantially older than QField, where snapping is straightforward to set up.

Overall I found Collector a straightforward app to use. Although the need to run the data through ArcGIS Online is slightly irritating and predisposes the process to additional errors, this is typical of a platform which is trying to serve a great variety of purposes across many industries.  The Collector app isn’t perfect (the absence of snapping is a serious defect in my opinion) and crashed a couple of times, but performance definitely improved when I downloaded the maps instead of using them through mobile-data/wifi. For anyone using Collector I would recommend keeping the number of layers, particularly raster layers, to a minimum by only using the layers you absolutely need in any given map. You can also improve performance by turning your rasters into tile packages and transferring them directly from your desktop to your mobile device. When you download your map to the Collector app it is also advisable to limit the ‘Work Area’ that is downloaded to the minimum that is necessary to complete your fieldwork, as this will speed up the download and performance generally.

QGIS QField

QField is the opensource QGIS version of ArcGIS Collector. It’s relatively new and still in the experimental stage, so may have some bugs, although these will be worked out as researchers find and report them. One great advantage of QField is that all data preparation takes place on your desktop GIS and there is no need to upload anything to an online account. Instead you prepare your data in QGIS desktop (including symbology, snapping and attribute fields), package it for QField using the QFieldSync plugin (Note that you must be using QGIS 2.18 for the QFieldSync plugin to function correctly) and then physically transfer it to your mobile device (with a cable). This eliminates many of the irritations of the ArcGIS Online transfer process, but does mean you need to pay attention to the QField online documentation about where you should store your project folder on your mobile device (I found this stage a little confusing, but more information will undoubtedly come out as people use QField more).

QField_interface_drone
QField interface showing the active layer (on the left) with menu above it and the data frame in the centre with a drone image of the Olynthos hills and surrounding surveyed fields (in red). The attributes of the yellow-highlighted field can be accessed by clicking on ‘198’ in the right pane.

On first look QField appears a little more confusing than ArcGIS Collector and the interface (see image above) is a little clunkier, but these are minor issues and will undoubtedly be resolved as development continues. Like ArcGIS Collector QField allows you to browse and view existing layers and create features. The workflows are typically clean and efficient, but unlike ArcGIS Collector you cannot edit the geometries of features that were transferred into QField from your desktop. You can only create and edit the geometry of new features. Although you can edit the attributes of existing features in QField, it is slightly disappointing that you cannot edit their geometries as it limits your capacity to edit previously created archaeological features based on new data obtained during your fieldwork.  It’s not an insurmountable problem, but it is irritating. Hopefully this will change with further development.

There are a couple of other minor issues with QField functionality. It’s frustrating that you cannot view features created outside of QField in vector layers set to ‘offline editing’. The advent of Android 7 has resulted in a bug in photo capture in the app (https://github.com/opengisch/QField/issues/133),  which is currently being resolved but prevented me from taking photos in the app.  However QField is still in the experimental testing phase and will undoubtedly improve with development. Given its stage development the quality of QField  bodes well for the future of opensource mobile-GIS.

GPS and accuracy

As our current survey fieldwork does not require better accuracy than the c. 2.5-5m provided by the onboard GPS I haven’t attempted to use these apps with an external GPS device. If necessary it would be possible to improve the accuracy with any one of a number of additional GPS devices, as described in this blog. This would undoubtedly add to the amount of equipment (and expense) involved and might provoke more compatibility and processing issues to occur, but would permit mobile-GIS to be used for much more detailed recording.

The future of mobile-GIS?

ArcGIS Collector and QGIS QField are very similar mobile-GIS apps and both permit a user to record data easily in the field, to a modest level of precision, and sync that data with their existing GIS using just a tablet or smartphone. Both apps significantly sped up data collection in the field and eliminated the need for additional data entry as records were transferred from total station or paper records to the desktop GIS. Currently ArcGIS Collector is the more developed and flexible platform, provided you can get access to it and are prepared to put up with the irritation of processing everything through ArcGIS Online. But QGIS QField is catching up fast, and has already surpassed Collector in some areas (notably the presence and ease of snapping) despite its experimental status. Although the occasional bugs might put some off QField, if you don’t have the resources for ArcGIS Collector, already have some familiarity with QGIS and are prepared to work around the occasional issue, QField is a great little platform that will only get better. In a few years’ time I can easily imagine using QField for preference if it continues to develop along its current lines.

Acknowledgements

I am grateful to the Peter Knoop and Caitlin Dickinson of the University of Michigan for creating the Olynthos ArcGIS Online account and initial ArcGIS Collector maps of the site. I am also grateful to the University of Michigan for access to the Olynthos field data, ArcGIS Online and opportunity to work with the Olynthos Project. I am particularly grateful to Zosia Archibald for suggesting my involvement in the project, to David Stone and Lisa Nevett for collaboration on the GIS, to Bettina Tsigarida of the Ephorate of Pella and to the Hellenic Ministry of Culture and Sports for permission to investigate the site and their collaboration on the project.

Additional References

For more information about the Olynthos Project see the Project website.

For fieldwork results to date see the forthcoming interim report Lisa C. Nevett, E. Bettina Tsigarida, Zosia H. Archibald, David L. Stone, Timothy J. Horsley, Bradley A. Ault, Anna Panti, Kathleen M. Lynch, Hannah Pethen, Susan M. Stallibrass, Elina Salminen, Sean Taylor, John Manousakis and Dimitrios Zekkos. Forthcoming. The Olynthos Project: Interim Report on the first three years of fieldwork (2014-2016). Annual of the British School at Athens.