This is not a beginners book on how to learn Python. This is the book you need, being the bookish type, if you use Python every day and would like to learn to finer details of how to use the standard modules effectively. This is a great book to just browse and start reading anywhere. I promise you that you will learn a lot.

I haven’t taken a whole lot of similar classes but enough to know which ones are a failure, and which ones are worth your time. The one for FME Desktop 2011 was one of the best I’ve taken. In fact, I couldn’t help myself and had to rave to the friends at Safe about it. So let me quote from my own email:

First of all, let me share with you that I found the FME Desktop class to be top-notch ! I have taken a few similar classes or workshops in the past and this one stands out. Not only did I get the impression that [the instructor] really knows her stuff never mind the occasional FATAL ERROR but the format of the class (30 minutes teaching, 10 minutes hands-on), plus the quality of the manual, made for a real good package that I would recommend to anyone interested in FME. Well done ! —  That said, I think FME is fantastic. I think I mentioned to both of you that I have pulled my hair out trying to wrap my brain around ESRI’s ModelBuilder more than once. FME makes more sense to me, is a lot more intuitive to my thinking and the work flows I deal with …

So, there you have it. If you haven’t taken a look at FME and deal with lots of ETL (extraction-transformation-loading) procedures, then this may be for you. Funny thing is (and I think both instructor and manual shared this with us), they say, that 90% of all FME transformations consist of shapefile >>> shapefile conversion e.g. reprojection, adding attributes… What does that say about ArcGIS if people use FME to transform shapefiles  ? I’ve certainly tried to use ModelBuilder more than once, and every time I found that is was much easier and straight forward to just write things out in Python. With FME, I felt that – while things can get very complex – the approach stays very clear and intuitive.

The table has the following headers:

UWI | NAME | MD | TVD | EWOFFSET | NSOFFSET | DIP | AZM | TVDSS | XPATH | YPATH | SURNAME | SURSTAT

and provides a row of data for each survey depth, with UWI(API) repeating for all rows belonging to the same well.

To read this into Python, I did this:

import csv

### Reference to Input CSV File
infile  = open('c:/temp/dirEXPORT.csv', "rb")
reader = csv.reader(infile)

rownum = 0
well = ""
Wells = []
points = []

### Read CSV File, Line by Line
for row in reader:

    if rownum == 0: # First line in file: read headers
        colcount = 0
        for column in row:
            print column,
            if column == "UWI":
                uwi = colcount
            if column == "XPATH":
                xpath = colcount
            if column == "YPATH":
                ypath = colcount
            if column == "MD":
                z = colcount
            colcount = colcount + 1
            rownum=+1

    else:

        print "\n"

        ### Check if row (line) is for the same well as row above
        ### if so, read coordinates from field and assign to xyz as list
        if row[uwi] == well:
            xyz = [row[xpath],row[ypath],row[z]]
            points.append(xyz)
            print row[uwi], xyz
            rownum=+1

        ### If row does not belong to the same well as the the last row,
### then append coordinate list and well name to Wells list, reset
### coordinates list (=[]), and assign the coordinates to the new well
        else:
            Wells.append([well,points])
            print "New Well"
            points = []
            well = row[uwi]
            xyz = [row[xpath],row[ypath],row[z]]
            points.append(xyz)
            print row[uwi], xyz
            rownum=+1
infile.close

Any remaining print statements allow me to watch what’s going on when the program is run. It’s my crude way of debugging code.
Once all the data is loaded into a list, I start some arcpy magic:

import arcpy
from arcpy import env
import os

env.overwriteOutput = True
env.workspace = "C:/temp"

cnt = 0

mypath = env.workspace
outputFile = r"wormtracks.shp"
template = r"c:\temp\template.shp"
fieldLength = 25

sr = arcpy.Describe(template).spatialReference

arcpy.CreateFeatureclass_management(mypath,outputFile,"POLYLINE","","","",sr)
arcpy.AddField_management(outputFile, 'API', "TEXT", "", "", fieldLength) # add API as new table field

rows = arcpy.InsertCursor(outputFile)
array_container = arcpy.Array()

for well in Wells[1:]: # Ignore first item in Wells list. Poor programming where an empty list is created.
    well_api = well[0] # First item in list is the API

    wellList = well[1] # 2nd item in list is a list of XYZ points

    for pt in wellList:
        point_object = arcpy.Point()
        point_object.X = float(pt[0])
        point_object.Y = float(pt[1])

        print pt[0]
        print pt[1]
        array_container.append(point_object) # I couldn't figure out the difference between array.append and array.add
        print point_object
        del point_object

    row = rows.newRow()
    row.shape = arcpy.Polyline(array_container) # Do not use arcpy.Multipoint which created a really messy polyline w/points ouf of sequence.
    row.API = well_api
    rows.insertRow(row)
    array_container.removeAll()

del row #unlock row
del rows #unlock table

del arcpy
del outputFile

ExecuteError: ERROR 000258: Output C:/temp/myfile.shp already exists
Failed to execute (CreateFeatureclass

That despite the fact that I had set

arcpy.env.overwriteOutput = True

(see me last post). So it’s a good habit to delete any references to cursors, e.g.:

del row #unlock row
del rows #unlock table
del arcpy

import gc
gc.collect()

>>> import arcpy
>>> sr = arcpy.Spatialreference()

Traceback (most recent call last):
File “<pyshell#4>”, line 1, in <module>
sr = arcpy.Spatialreference()
AttributeError: ‘module’ object has no attribute ‘Spatialreference’

>>> sr = arcpy.SpatialReference()
>>>

 >>> fc = "c:/temp/myfile.shp"
 >>> desc = arcpy.Describe(fc)
 >>> print desc.spatialreference.name
 NAD_1927_StatePlane_Texas_South_Central_FIPS_4204
 >>>

>>> print desc.spatialreference.name
NAD_1927_StatePlane_Texas_South_Central_FIPS_4204
>>> print desc.SpatialReference.name
NAD_1927_StatePlane_Texas_South_Central_FIPS_4204
>>> print desc.SpatialReference.name
NAD_1927_StatePlane_Texas_South_Central_FIPS_4204

fc = "c:/temp/dirSurvey.shp"
desc = arcpy.Describe(fc)
sr = desc.spatialReference
arcpy.CreateFeatureClass_management("C:/temp/","newFC","POLYLINE", "", "", "", sr)

arcpy.CreateFeatureclass_management("C:/temp/","newFC","POLYLINE", "", "", "", sr)

from arcpy import env
env.overwriteOutput = True # instead of overwriteoutput which would be wrong

>>> dir (arcpy) # or dir(arcpy.env)

I was tasked with going through a directory tree (sound familiar ? I have been do this sort of thing a lot lately), finding all PDF files and merging them into one large PDF file. The first thought was: what kind of Adobe product do we need for that. Then, I said: let me try that in Python. That in fact, turned out to be very straightforward. Simply download pyPdf, and take a look at the example there. No needc for Adobe.

import pyPdf
import os

startDir = "c:/temp"
os.chdir(startDir)

fileList = os.listdir(startDir)
output = pyPdf.PdfFileWriter()

for item in fileList:
if os.path.splitext(item)[1].upper() == ".PDF":
pdfDocument = os.path.join(startDir,item)
input1 = pyPdf.PdfFileReader(file(pdfDocument, "rb"))
for page in range(input1.getNumPages()):
output.addPage(input1.getPage(page))

outputStream = file("MyNewOutput.pdf", "wb")
output.write(outputStream)
outputStream.close()

Just replace “c:\temp” with the directory you want to pull the PDF from and you’re in business. If I combine this with the recent script I wrote to explore directory structure, you can drill down through the directory tree searching for and merging PDF files.

The idea was to type in a starting directory and a name for a text file, after which the script recursively drills down through the directory tree, printing file names, examining file extensions, counting files and file size, and then spits out totals, plus – for a visual aid -  prints a quick and dirty histogram. So let me know what to improve next time around.

from __future__ import division

### Cobbled Together by Arne, July 2011
### using the MyOutput() bits by xiao, from
### http://tech.xster.net/tips/python-log-stdout-to-file/

import os, sys

class printToFile():
''' directs print output (stdout) to textfile'''
def __init__(self, logfile):
self.stdout = sys.stdout
self.log = open(logfile, 'w')

def write(self, text):
self.stdout.write(text)
self.log.write(text)
self.log.flush()

def close(self):
self.stdout.close()
self.log.close()

def dictHistogram(extCount):
''' Creates simple histogram based on key-value entries in
file extension/frequency dictionary (extCount) '''
vcount = 0
kcount = 0
for k,v in extCount.iteritems():
vcount = vcount + v

for k,v in extCount.iteritems():
#print v
share = round((v/vcount) * 100,2)
print v, "\t", "File Type ", k, "\t", int(share)*"#", share, "% of total"

def displayFileInfo(entryPath):
'''Displays file name, size when exploreSub() encounters an entry
that is not a directory. New extensions are added to the extensions
list, files are counted in totalFiles, and file size is added up
in totalSize'''
global extensions
global totalSize
global fileCount
global totalFiles
global extCount

print "\t",os.path.basename(entryPath)
ext = os.path.splitext(entryPath)[1].upper()

totalSize = totalSize + os.path.getsize(entryPath)

if ext not in extensions:
extensions.append(ext)
extCount[ext] = 1

else:
a = extCount[ext]
a = a + 1
extCount[ext] = a

def exploreSub(dirEx):
''' Drills down into a file tree, starting with dirEx. If an empty
directory is encountered, function breaks from loop, if a file is
encountered, the file's path is added to a list of files to examined
later, and if a non-empty directory is encountered, exploreSub is
recursively drills down to the next level. Once all directories have
been explored, the files in the list are examined one at a time.'''

global files
global fileCount
print
print dirEx

if os.listdir(dirEx) == []:

print "Is an empty directory."
print

else:
for entry in os.listdir(dirEx):
entryPath = os.path.join(dirEx,entry)
if os.path.isdir(entryPath):
print "\t", entry, "<Dir>"
else:
displayFileInfo(entryPath)

for entry in os.listdir(dirEx):
entryPath = os.path.join(dirEx,entry)
if os.path.isdir(entryPath):
try:
exploreSub(entryPath)
except:
Print "Unable to open ", entryPath

extensions = []
files = []
extCount = {}
fileCount = 0
totalFiles = 0
totalSize = 0

start = raw_input("Enter Directory: ")
log = raw_input("Enter Name for Logfile: ")

sys.stdout = printToFile(log)
exploreSub(start)

print
print "Total Number of Files ", totalFiles
print "Total File Volume ", round(totalSize/1048576), " MB"
print "File Types Encountered: "
print extensions
print

dictHistogram(extCount)

The GIS department I’m part of has been tasked to build an enterprise GIS using ArcGISServer, ArcSDE, SQLServer 2008, and MS Silverlight. Moreover, we are working on a Master Data Management Solution that will allow data from multiple departments to be available through the Silverlight powered web app.

For me, the challenge, therefore, has been and continues to be two fold:

1) Brush up on many aspects of ESRI technology:

ArcGIS Server, ArcSDE, Python. Silverlight…

2) Learn about a whole range of oil/gas industry software applications:

IHS Petra, SMT Kingdom, NeuraSection, Landworks, Merrick/RIO, the list continues

and how they might become part of the MDMS using NeuraDB, Volant, FME…

I guess you could say we got our work cut out for us.