Monthly Archives: January 2010

“Cropping” Shapefiles – Clipping Features

I have gotten a little carried away teaching myself Python, wrapping my brain around OOP and designing classes. But I’ve had relatively little opportunity using all that Python for what I intended it to help me with – Geoprocessing in ArcGIS. So I decided on a little review of what I learned a while back and apply that to a very common task in my “day job”.

What I spend about 80% of my GIS time doing is making pretty maps for engineering reports. Typically, these are based on SHP data for infrastructure (street, highways, interstates), hydrology (streams, rivers, lakes), administrative boundaries (city, county), geology, and project related information (e.g. boreholes, property boundaries, building foot print).

For North Texas projects, I was using GIS data from the North Central Texas Council of Governments (NCTCOG). But for projects in other regions of the state, I have been using ESRI data, USGS-NHD data, TNRIS data (especially for Texas geology), and data from other local sources, e.g. CAPCOG, for the Austin, Texas, region.

The “problem” with some of the ESRI data, apart from its datedness (When you’re working on construction related projects, it doesn’t help you that entire subdivisions or brandnew toll roads are missing from your data sets.), has been that it comes in SDC (Smart Data Compression) format. Which is great for cramming nationwide data onto DVDs but a pain when you end up adding Alaska roads to a Tarrant County, Texas, project MXD. Hey, wait, hear me out – of course, I know that this is not a “problem”. I’ve just been too lazy (busy, both) to fix it since half the time I can manage with only SHP files instead of SDC.

So how to “crop” those large files down to the region of interest ? Python ! While you can perform geoprocessing on feature classes in a geodatabase, you’re limited when it comes to geoprocessing with feature classes in SDC files. So one way out is to extract the feature class you need in ArcCatalog. I was amazed at the compression of SDC when I saw that extraction of one feature class from a 300MB SDC file created a 1.34 GB SHP file and took 34 minutes. That begs the question whether working with only the SHP you need is even more efficient (faster) than using the SDC file.

But I needed only a small portion of the SHP file – enough to cover a 15-county area. So I wrote this little micro Python script:

import arcgisscripting
gp = arcgisscripting.create()

    # Set the workspace
    gp.Workspace = “c:/GeoProc

    # Process: Clip the det_riv shapefile to the extent of the metro_county shapefile and name it metro_riv
    gp.Clip_analysis(“det_riv.shp”, “metro_county.shp”, “metro_riv.shp”)

    # In the event of an error, print messages
    print gp.GetMessages()

This created a 5MB shapefile with only rivers in the 15-county metro area I do most of my work in. Okay, this wasn’t magic, and a dozen lines of code are barely programming. But it worked, and I am glad I finally got around to using some Python on something I’ve been meaning to try in a long time.

1 Comment

Filed under Uncategorized

ESRI = Future of GIS ?

When I first started working with ArcGIS, I didn’t give much thought to the “makers” of it. ESRI made software just like other companies, and having worked with Mapinfo years ago, I figured they weren’t lonely players in the field of GIS.

Later I started researching “Open Source GIS” and found what seemed like a parallel universe of Open Source GIS bundles and applications. There is a lot of development going on with GIS software that is totally free.  So why choose ArcGIS ?

Obviously, ESRI software has become something of a standard, and for most jobs it’s a minimum requirement. If you want to work in GIS, you need to know how to work with ESRI products. When I thought about returning to my Mapinfo “roots”, I noticed it had retreated to something of a niche market (please correct me if I’m wrong!).

I’ve always liked the Open Source idea. It took me a long time to warm up to the idea of using Linux. But when I tried it, I quickly became convinced that the Era of Windows would end, and Google has certainly given a huge boost to the Open Source Universe.

There are a lot of things you do with ArcGIS that you can probably do just as well and easily with Open Source GIS. When it comes to other functionality, e.g. I have heard people mention Spatial Analyst, maybe you can’t. So when I decided to pay for an EDN license, it was because I figured I needed the ESRI specific skills to work in GIS.

I’ve come across two interesting items today that make me wonder if ESRI will continue dominating the GIS market, and which encourage me to continue following the happenings in the Open Source world. In my last post, I linked to James Fee’s blog, where I read a few interesting things, e.g. :

The legacy they [ESRI] may be fighting is the COM framework on which most of ArcObjects is based. At past Developers Conferences I’ve asked if there are plans to migrate their libraires to pure .Net. (Dennis Geasan)

Quad core Win7 workstation for ESRI? Remember, no multithreading, no multicore support…and you’re stuck in 32 so don’t bother with anything more than 4gigs of RAM, my friend. (Archie Belaney)

That after I just bought a new PC, quad core and 6GM RAM to run ArcGIS Server.

And then this page from last year, where ESRI bailed from the WMS Server Shootout against Mapserver and Geoserver.

Now, none of this makes me enjoy working with ArcGIS less than before. But it brought up the question where the mainstream in GIS will be headed in the future.

Leave a comment

Filed under Uncategorized

ArcGIS 9.4 Beta 1, no, wait! – 2, or ArcGIS 10 ?

My inbox has been filling up with emails from ESRI about the Beta testing program for ArcGIS 9.4 (now 10). So I finally downloaded the new version. But I was wondering if I can keep my 9.3 installed while I take a look at 9.4. Maybe I didn’t have to download the 3GB installer after all because apparently you cannot. I found the answer on James Fee’s blog in a discussion thread that included some very interesting comments about ESRI’s aging technology.

1 Comment

Filed under Uncategorized

First Web Application with ArcGIS Server

Building a first web app with ArcGIS Server was meant as a test to see if I had installed and configured everything correctly. I have AGS running on Windows 7, and my desktop client is running Windows Vista with ArcGIS 9.3 and IE8.0. Since I am entirely new to AGS and its capabilities, I didn’t really know what to expect. So I stuck with the instructions in the EDN documentation library for first steps with web apps.

I created a simple project (.MXD) in ArcGIS Desktop using only four (4) shapefiles for North Texas –  counties, cities, highways, and hydrology, data I downloaded a long time ago from the North Central Texas County of Governments GIS (NCTCOG) website. Something simple, yet familiar to experiment with. Once my MXD was there, I started ArcGIS Server Manager, selected “Publish GIS Resource”, added my Project1 and published it. That was the easy part.

I spent the rest of the time with Project 1 trying to figure out how to access this resource on the server with ArcCatalog or Internet Explorer on the client. In the process of getting this to work, I came across a number of issues that may have been causing error messages. In the end, I am afraid, I am not sure what finally solved the problem. But I finally figured it out, and I love the results.

When AGS informed me that service Project1 was running, I tried http://<mylocation>/<arcgisserver>/services on the client to pull up the web app. But I got an error that “the connection could not be made”.

It appears that often problems with connecting to AGS from a client have to do with DCOM – Distributed Component Object Model. So a first step was to check whether both server and client had DCOM enabled.

Then, I had to check R/W permissions for the SOC/SOM accounts on the server. Those appeared to be in order. And just in case, because someone somewhere in a discussion thread suggested it, I deleted and added again the SOC/SOM/Web accounts. But that didn’t help.

At some point, the error message had changed to :

Access Denied: Either the SOM service on the machine is not started or the user attempting this connection is not a member of the agsuser or agsadmin group and cannot be granted access to the SOM.

This sounded like the client was “seeing” the server but had no permission to “talk”.

On the ESRI website I found another Bugfix. Again, I checked whether SOM/SOM had remote launch and activation permissions set correctly, and they did.

Finally, I went back to agsusers on the server (see my post on install/post install), and it occurred to me that the client wasn’t part of the agsusers group. How to fix that since only local users (accounts on the server) show up as choices to add to the group. Going through some of the ESRI pages again, I finally figured out that I can add an account on the server that is identical to the one used to log into the client.

In general, the list of accounts you add to the ArcGIS Server user and administrator groups will depend on what clients you anticipate will connect to the server. When connecting to a windows GIS Server, each operating system account from which ArcGIS Desktop is run will need to be added to the ArcGIS Server users group if you want that client to access the GIS server. Additionally, each Web application you create can connect to the GIS server through a particular operating system account. Each account your Web applications utilize should be added to the ArcGIS Server users group as well.

Once I added this to the agsusers group, I was able to see my Project1 service running on Internet Explorer. Again, many lessons learned !

1 Comment

Filed under Uncategorized

ArcGIS Server on Windows 7…(The Installation)

Finally ! I have ArcGIS Server (AGS) Up and Running. It’s been a “circuitous” installation, and I am looking forward to using AGS. Here are the milestones.

1) Where to install ?

I attended an ESRI session about AGS last week at the DFWMAUG January Conference and heard quite a bit about AGS capability and server design etc. I had terms such as SOC and SOM buzzing around my head… by now, I know a little more about them. The simples install – especially for development purposes, I learned, was to have ArcGIS Desktop and AGS (and hopefully the other EDN products) all installed on the same machine. But what machine ? I wanted one separate from the one I do all my non-GIS work on. So I bought another PC.

I ended up with a Gateway DX4831-01e (Intel® Core™ i3-530 processor (2.93GHz), 6GB DDR2 RAM, 1TB Serial ATA HD, and Microsoft Windows 7 Home Premium Edition 64-bit).

Little did I know about Windows 7 (although I had made sure that AGS would run on it), less did I know about 64 bit.

2) Preparing for Install.

There are a number of prerequisites and steps in preparing your machine for AGS installation. ESRI’s ArcGIS Server Installation Guide (here and here) helps cover most them. But it includes no info for installing on Win7. Instead, I found one helpful page about AGS onWin7 here.

First I made sure that the required features of Internet Information Services (version 6 on my machine) were turned on, as shown here. In Win7, there were some differences between “checkable” features under Security. It later turned out that the only way to get things to work under Win7 is ti simply turn on ALL features of IIS6.

Working with 64-bit Windows7 oi the next consideration. ArcGIS Server is 32-bit and 64-bit Windows needs to be told to be friendly with AGS. Follow steps 1-4 in ESRI’s AGS Installation Guide.

Step 2 caused an error with finding adsutil.vbs. – I fixed this by turning on  IIS6 Scripting Tools. Which was followed by a permission denied – Error Number 70(0x86) – because I failed to run cmd as administrator. You do so by doing Start > Run > cmd > Ctrl/Shift/Enter instead of Enter. That fixed the problem.

Step 3 was no problem. Step 4 was slightly different in Win7. Goto ControlPanel>System/AdminTools… Advanced Settings>Enable 32-bit… It should say “TRUE”.

Next, I installed Visual Basic 2008 Express and Visual C++ 2008 Express, and Web Platform Installer which includes SQL Server Express (I don’t think that was a requirement).

3) Installation

Time to pop in the DVD. Have your authorization file handy, and click through screens about SOC and SOM and WebServices. This was the only straight forward step (on the 4th try or so…).

4) Post Install

Start with this page here to add yourself (user) to the agsadmin Group. This was different on Win7. Control Panel > System&Security > AdminTools > ComputerManagement > BLANK. You’re supposed to see Groups and Users here. But I didn’t. Turns out that my new PC running Win7 Home didn’t provide this option. Time to upgrade to Win7 Professional. Hopefully this wasn’t $89 wasted. — Once Wind7Prof was installed, I proceeded to Groups > agsadmin. Right Click > Add to Group > Advanced > Find Now = here you select your user name.

5) Test Drive

AFter restarting the computer, I see ArcGIS Server in my Start menu, and find ArcGIS Server Manager. I can start it but when it comes time to login using Domain/User (in my case arcgisserver/myname) and password, nothing happens. So there remains once more thing to take care of and I found the solution here. Go to Admin Tools (see above) > Local Security Policy > Security Options  and make sure that 128-bit Encryption Requirement is unchecked for Network Security: Minimum Sesssion… clients and Network Security: Minimum Sesssion… server.

Now, you you should be able to start AGS Manager and log in successfully. – I hope this helps. It took me the better part of a day to figure it all out.


1 Comment

Filed under Uncategorized

Getting Started with ArcGIS Server

This week, I received my EDN package from ESRI. Wow, this is exciting !

This bundle is so loaded with DVDs – it’s hard to decide where to start. For now, I’ve settled for ArcGIS Server. I’ll be looking for some very basic introduction to the whole ArcGIS architecture (beyond the desktop environment) and ArcGIS Server in particular. So far, I’ve found the ArcGIS Server Developer’s Guide, one of the many ESRI documents available, and the ESRI online help for ArcGIS Server. If you know any other good starting points, please let me know. To be continued…

Leave a comment

Filed under Uncategorized