Querying Data from ArcGIS REST Services Using Python and ogr2gui

Purpose

The purpose of this knowledge base document is to outline how to utilize a python library to query data from an ArcGIS REST service.

ArcGIS Server queries are typically limited to 1000 features at a time. This can pose a problem if one is looking to say, query the parcel data from a large county. This method automatically queries a defined REST service in batches of 1000 features, concatenates the batches, and returns all off the features in a .geojson file. That .geojson file is then converted directly into an Esri Shapefile through the use of the ogr2gui application.

Note - There is an excellent alternative to this method which is to use data pillager (linked below). However, it is dependent upon arcpy and may struggle with large datasets. The method described below is platform agnostic and only limited by the amount of RAM (and time) you have.

https://github.com/gdherbert/DataPillager

Scope

The intended audience for this document is a GIS Analyst with basic programming skills.

Prerequisites

  • Basic knowledge of Python and scripting.
  • 64-bit Python 2.7.9 or higher.
  • PIP Python package manager.
  • ArcGIS REST Query Python Utility.
  • ogr2gui application.
  • 7zip, or similar.

Responsibilities

This document should be used by any GIS Analyst who needs to query a large amount of features from an ArcGIS REST Service.

Procedure

  1. Open the test Python script located located at I:\GIS\Development\Python\Scripts on MosaicGIS. You should see a heavily commented version of the script similar to the below.
import json
import arcgis

from arcgis import ArcGIS
service = ArcGIS("http://staging.storymaps.esri.com/arcgis/rest/services/Gettysburg/GettysburgTroops8/MapServer")
##Defines the specific REST service that we want to target
#descriptor = service.get_descriptor_for_layer(1)
##Returns the JSON descriptor for the layer.
##This tells you things like what fields are in the layer, what sort of geometry it contains, etc. 
#field_list = service.enumerate_layer_fields(1)
##Returns a list of the field names in the layer.
query = service.get(35, count_only=False)
##Retrieves a single, defined layer from a web service. 
##If count_only=True, we return a simple count of the number of features in the layer you're querying.
json_query = json.dumps(query)
##Creates a new variable, 'json_query', and defines it as a JSON object.
f = open("union_move.geojson", "w")
f.write(json_query)
f.close()
##Validates our geoJSON and either writes it to a file if successful or returns an error message if unsuccessful.
#print query
  1. This script carefully outlines what each field does and what inputs are necessary to make it work.

  2. Alter the inputs to point to the REST service you are looking to query, query the correct layer, and return the correct file name.

  3. Open the ogr2gui application which should look like so.

  1. Select the .geojson file generated by the query in the Source field. Then choose where you would like to save the resulting file within the Target field. Ensure that the Source format is set to geoJSON and the ‘Target’ format is set to ESRI Shapefile.

  2. Use 7zip, to compress the resulting files so that it may be imported into ArcMap, QGIS, ArcGIS Online, or other.


References

Quality Control Checklist

Import the resulting Shapefile into ArcMap/QGIS and ensure that you are able to view the attribute table. If an error is received ensure that there are no column names that include special characters. If a column name does include special characters it can be altered or removed entirely within editing mode.

Definitions

REST

  1. [Internet] Acronym for Representational State Transfer. An architecture for exchanging information between peers in a decentralized, distributed environment. REST allows programs on different computers to communicate independently of an operating system or platform by sending a Hypertext Transfer Protocol (HTTP) request to a uniform resource locator (URL) and getting back data in some format—for example, XML, or inside a URL. REST is used in Web services.

GIS Dictionary: http://support.esri.com/en/knowledgebase/Gisdictionary/browse

Was this FAQ helpful?
FAQ details:
Published date: 11/10/2017 4:39PM
Last updated: 11/10/2017 5:13PM (Chris Rice - crice@interdev.com)
Author: Chris Rice (crice@interdev.com)
^ Top of Page