I Get By With A Little Help From My Friends

Talking to a friend the other day it occurred to me how fantastic it would be if Vr could import and display OpenStreetMap data.  Nothing fancy mind you but maybe create an image that you could use as a background for a quick check of the surrounding area or even flight prep.  Wow that sounds like a lot of work unless of course someone else has already done it.  One of the many beauties of Python is the number of fantastic libraries out there if you just spend a little time tracking them down.  If you take that a step farther and are willing to pass a little of the responsibility outside python using system calls or batch files the possibilities are nearly endless.  Let's just say I can now display an OpenStreetMap map image in Vr and it only took about 90 lines of python.  Here are a few of the major helpers besides the map site which is so worthy of support, contribution, and praise.

I know I have mentioned it before but I can't say enough about pyproj which I use to convert my local coordinates to geographic for sending out to various places.

Once I have the corners in Lat / Long the rest will be done by a batch file which is about the only thing my python script creates.  It could be done in other ways but I liked the idea of creating something that could stick around and be easily modified for use again.  So the batch file next makes a call to the openstreetmap.org API for grabbing the map data.

Once the OSM file is in house it will be converted to a bitmap using Maperitive.  I only use such a tiny fraction of the capabilities of this amazing program, but for my purposes using it's command line version with a custom generated script will fit the bill perfectly.

The original bitmap is created in Web Mercator and can also throw in a TFW and KML file for generic use or display by anybody if you want to send them along.  I however want to display it behind my map in a local coordinate system in whatever system I choose so the last step is to re-project the image using GDAL.

Is this the most advanced, professional, elegant way of doing it?  Of course not, you obviously don't know me very well.  As usual it is something you can hack together in just a short time with minimal duplication of effort that gets the job done.  Hooray for real programmers!!!

Don't forget to support your favorite projects with contributions or cash when possible.

Note to self: Step backward through LiDAR when removing, and repr ( round ( a number )) is awsome

First, apparently I never noticed this before but while trying to step though a bunch of LiDAR points to remove a certain class it apparently changes something dynamically (like the point number in the buffer).  I noticed this because I was doing a

    for PointNum in range(Punt.GetCount()):
        if Punt.Cla(PointNum) == Whatever:
            Punt.DelPunt(PointNum)

To get rid of a certain type of point, in this case based on class.  What I noticed though was that each time it ran there were a bunch of points left over (half??).  I decided that even though I wasn't doing a record until the loop was done, the point number in the buffer must change as points are deleted.  So what to do?  How about starting at the last point and removing them from the top end of the buffer so as you work backwards the previous point numbers would not have changed.  I came up with

for PointNum in range(Punt.GetCount(),-1,-1):

and it seems to work fine.

Second and totally unrelated I needed to rebuild a tile naming structure for a local dataset.  There is a tile scheme that is based on 5000 foot tiles that is easy enough to get the names of.  The problem is that there is also a subset of 1250 foot tiles that starts from the corner of each 5k block but the names are based on 1000 foot increments.  When you digitize a random point in a block you can't necessarily round down to the lower 1000 because they just truncate the thousands from a rounded 1250.  Ok hard to imagine but the corners would be 0, 1250, 2500, 3750 so a number 3749 would fall in a 2500 block and truncate to 2000 meaning the tile would have 2 in the name, not to mention dropping a million off one of the coordinates.  Now for the why I love python part!

repr(round(1773691/1250)*1250)[1:4]

yields '772' just like is necessary to build the proper tile name. 

Note to self: Find a reason to use geopy

Full disclosure; This has nothing to do with anything except that I was playing with a really cool module today and need to make sure I remember what I just learned.  The module is geopy (which by the way I installed using pip for the first time ever ) which can be used to geocode natural language addresses using a variety of APIs.  Don't know when, how, or why I'll use it but I'm certain I will.

Info can be found at https://pypi.python.org/pypi/geopy
Doc at http://geopy.readthedocs.org/en/latest/

Here is what I just did (using my current address)

>>> import geopy
>>> geopy.geocoders.GoogleV3().geocode("4090 weaver ct s 43026")
Location((40.035917, -83.144145, 0.0))
>>> geopy.geocoders.GoogleV3().geocode("4090 weaver ct s 43026").raw
{u'geometry': {u'location': {u'lat': 40.035917, u'lng': -83.144145}, u'viewport': {u'northeast': {u'lat': 40.03726598029149, u'lng': -83.1427960197085}, u'southwest': {u'lat': 40.03456801970849, u'lng': -83.1454939802915}}, u'location_type': u'ROOFTOP'}, u'formatted_address': u'4090 Weaver Court, Hilliard, OH 43026, USA', u'place_id': u'ChIJpUOLF-iTOIgRT-Vqu5rHhLo', u'address_components': [{u'long_name': u'4090', u'types': [u'street_number'], u'short_name': u'4090'}, {u'long_name': u'Weaver Court', u'types': [u'route'], u'short_name': u'Weaver Ct'}, {u'long_name': u'Northwest Industrial Complex', u'types': [u'neighborhood', u'political'], u'short_name': u'Northwest Industrial Complex'}, {u'long_name': u'Hilliard', u'types': [u'locality', u'political'], u'short_name': u'Hilliard'}, {u'long_name': u'Norwich', u'types': [u'administrative_area_level_3', u'political'], u'short_name': u'Norwich'}, {u'long_name': u'Franklin County', u'types': [u'administrative_area_level_2', u'political'], u'short_name': u'Franklin County'}, {u'long_name': u'Ohio', u'types': [u'administrative_area_level_1', u'political'], u'short_name': u'OH'}, {u'long_name': u'United States', u'types': [u'country', u'political'], u'short_name': u'US'}, {u'long_name': u'43026', u'types': [u'postal_code'], u'short_name': u'43026'}, {u'long_name': u'1197', u'types': [u'postal_code_suffix'], u'short_name': u'1197'}], u'partial_match': True, u'types': [u'street_address']}

You gotta love a good library

The Columbus Metropolitan Library is arguably one of the best libraries in the nation.  I use it so much I have had my library card memorized for over 25 years.  The idea that there is information and services there at my fingertips any time I want is fantastic.  I was thinking about this yesterday when I started pulling functions out of PyVrGeom().  Libraries (modules) can be everything from little bits of code snippets that you want to reuse over and over to complex collections of classes that you could only dream up but never realistically code yourself. 

For example PyVrGeom is just a collection of math that some would find easy enough to code, but why do so when somebody has done the heavy lifting and offers you the calls to do it with some preexisting code.  In this case very professionally created and presented, but it is just as easy to hack together some snippets that just save you typing.  For example when I was working on a KML export routine I didn't need to learn something complex and universal (which libs probably exist), I just knew what it looked like and wanted to replicate those entities by passing along a few basic parameters.  In this case I slapped together some code to create the KML entities, but at the same time the coordinates needed to be in geodetic coordinates so I had to grab a projection library that someone way sharper than me had already put together. Then it struck me that a KMZ is just a file zipped using the standard algorithms so if I import that one I can just as easily write both KML and KMZ files.

I'm not going to offer any code this time just another reminder that there are some good reasons to install python if you are going to do some scripting.  Here are some of the libraries that I really enjoy and use often.

pyproj - for converting local coordinates to global (or from one to another).
liblas - for all things LiDAR related.
zipfile - for reading and writing compressed files.
xlrd/xlwt - for directly reading and writing Excel spreadsheets.

Along with some of the standard system libraries that you can find a million uses for like sys, os, time, and math.

And here is a tip I'll add for free, check out http://www.lfd.uci.edu/~gohlke/pythonlibs/

It's all part of the process (or subprocess as the case may be)

On occasion I make the case for installing python in order to extend the usefulness of the environment in Vr, or at least getting hold of some of the most useful standard libraries.  Well here is another good reason.  This is just a snippet that I will use later but I figure this is as good a place as any to keep track of it.

Before too long I want to write a script that will depend on running an external process, then interacting with the results of the process (no spoilers yet).  In order to do that I'm going to make something similar to the following call to process some files leaving the results in the working directory. The research I did here was to make sure I had a way to not proceed until the original process was complete.  Later I'll need to make sure I can run a shell command and capture the resulting output but this will do for now.

import subprocess
process = subprocess.Popen('lasboundary -i *rgb1.las -otxt')
process.wait()
print process.returncode
print '\a'

the last line is just to beep so I knew when it was done.

Reminder snippets - liblas and reading laz files

The other thing that I used to do with this blog is post things that I wanted to remember.  At one point the Vr forums were a good place to do this because it also gave an opportunity to share and collaborate.  Since that doesn't really seem to be happening any more I suppose I could drop things here and then come back to them later if necessary.  Yes I could just do this in a document but who knows maybe a new opportunity for some kind of sharing community will pop up again.

In this case I just want to remind myself that if Vr never adopts the .LAZ LiDAR compression format for reading, it would still be possible using liblas. This is simple a little interactive session that shows it is theoretically possible using my own data.

There may indeed be other better ways, but this seems to work fine.

>>> import liblas
>>> f=liblas.file.File(r'c:\tmp\N1915250.laz',mode='r')
>>> f

>>> p=f.read(100)
>>> p.classification
5
>>> p.x
1919853.35
>>> f.header.compressed
True
>>> f.filename
'c:\\tmp\\N1915250.laz'

Projecting some thoughts about pyproj.

So I've been playing around with pyproj a little lately and have come up with a couple new thoughts.  As always I'm typing as I go so not everything is a clean example of good code or procedure, just something I've run as a test of my thoughts.

First, the application; below is showll.py which when run displays the lat long of the current cursor position in the DspMsg0 area (the top long line right below the command area).  I decided to do this as an app so it is running in real time and clicking the buttons just performs specific tasks.  Speaking of the buttons, B1 or the left mouse button prints the current position in the local coordinates, long lat, and notes the projection used. Something like

1369447.29077 471184.7857 (-84.66858719560254, 40.93954025528449) epsg:3753

In this part of the world we primarily deal with two projections so I kept it simple and just used B2 or the right mouse button to toggle between the two.  It would have been easy enough to bring up some kind of selection dialog, but for now I only need the two. The current projection is also displayed in the DspMsg area.

I recently discovered why I always had to send and receive meters even though the EPSG code clearly specified at foot based projection.  It was by design so there was never any question what was expected or returned.  The answer was always meters, however the code has since been modified to include an init keyword

preserve_units=True 

which handily enough, does exactly what it says. I understand the value of the consistency, but appreciate the option. I have also discovered in testing that you can take a proj4 definition and paste it straight into an object init function in quotes and get the exact same functionality as using the EPSG codes.  For example...

ohs=pyproj.Proj('+proj=lcc +lat_1=40.03333333333333 +lat_2=38.73333333333333 +lat_0=38 +lon_0=-82.5 +x_0=600000 +y_0=0 +ellps=GRS80 +to_meter=0.3048006096012192 +no_defs' )

Well in any case here is the app, it isn't worth much except by way of example, but pyproj itself is really handy when working with things like flight prep or control layouts when you want to give somebody geodetic coordinates to work from.

print 'showll modified 12:17 PM 11/14/2012'
# Show Longitude Latitude of current position.
'''
Copyright 2012 Dennis Shimer
Vr Mapping copyright Cardinal Systems, LLC
No warranties as to safety or usability expressed or implied.
Free to use, copy, modify, distribute with author credit.

Only two projections are supported. Both are NAD83(HARN) / Ohio (ftUS).
EPSG:3753 for North Zone
EPSG:3754 for South Zone
Button 2 (right mouse) toggles the projection, Button 1 (left mouse)
will print out the current position in local coordinate and it
Longitude, Latitude decimal degrees.
'''
MYAPPID = 10011

Gui = PyVrGui()

import pyproj,sys

global Projection

Projection=pyproj.Proj(init='EPSG:3754', preserve_units=True)
Gui.DspMsg('OH S EPSG:3754')

def MyAppDigCB (x, y, z, key, id):

 global Projection
 Gui.DspMsg0(repr(Projection(x,y,inverse=True)))
 if (key == 1):
  print x,y,repr(Projection(x,y,inverse=True)),Projection.srs.split('=')[1]
 elif (key == 2):
  if Projection.srs.count('3754'):
   Projection=pyproj.Proj(init='EPSG:3753', preserve_units=True)
   Gui.DspMsg('OH N EPSG:3753')
  elif Projection.srs.count('3753'):
   Projection=pyproj.Proj(init='EPSG:3754', preserve_units=True)
   Gui.DspMsg('OH S EPSG:3754')
 elif (key == 11):
  MyApp.Close ()
  MyApp.StopRunning ()

def MyAppKeyCB (key):

 MyAppDigCB (x,y,z, key, 0)

try:
 MyApp = PyVrApp (MYAPPID, "ShowLL")
except:
 print "Could not start application:", sys.exc_info()[0]
 
try:
 Mk = PyVrMenuKeys ("ShowLL", 10)
 Mk.SetMkLabels ("ShowLL","1 Print pt","2 Tog N/S", "3", "4", "5", "6", "7", "8", "9", "*", "0", "# Quit")
except:
 print "Could not create menukeys:", sys.exc_info()[0]

try: 
 MyApp.SetDigCB (MyAppDigCB)
 Mk.SetKeyCB (MyAppKeyCB)
 MyApp.KeepRunning ()
 
 Mk.Delete ()
except:
 print "Exception during application:", sys.exc_info()[0]
 Mk.Delete()
 MyApp.Close ()
 MyApp.StopRunning ()

How about one more bonus snippet with regard just switching from on projection to another. Thanks to Proj4 and pyproj, this is just how easy it can be.

>>> ohs=pyproj.Proj(init='EPSG:3754', preserve_units=True)
>>> ohn=pyproj.Proj(init='EPSG:3753', preserve_units=True)
>>> pyproj.transform(ohs,ohn,1759595.61,661026.83)
(1759551.3008025475, 54039.04746006191)

Time out

It is that time of year when I begin to pursue my second job as middle school soccer coach, and high school referee (and parent spectator). Because of this, I'm not even going to think about posting until late October.

Life Outside Vr Wrap Up (almost)

Before going back to writing functions for Vr let's wrap up a couple of things. First binary files, I have recently begun to like the idea of persistent parameters for several of my programs. I really need to create some basic generic parameter code to paste in. In the end I'll probably settle on ascii files with keywords, or at least comments. But what if you want to use binary, or what if you want to read a binary file that you know (or could guess) the structure of. An example is the old .COO coordinate files used by orientp3 in pre-Vr days. We still use it and often want to work directly with the .COO files for various things. But they are binary, here are some basics working with something similar to the parameter file we used before. This will be plain python, you can copy and paste this, download testbinary.py from my source code website, then either run it from a command line like
python testbinary.py
or open or paste it into pyedi and run it inside Vr. The only requirement is that you will need to have python installed because it uses two standard modules. By this time I am going to assume some python knowledge and ability to read the code. I am thrilled to answer any questions that come up, but I will post this code in a very plain, line at a time manner rather than using multiple methods on a single line. I'm also going to use my favorite method of accessing binary data since as is common in python there are probably dozens of ways of doing it. You could probably do this in about 8 lines but the point is to explain what is going on. I'll also assume that anyone interested in this would take the time to look up the modules listed to see how many cool things they will do... So here goes

If you were to run this script 3 consecutive times it would give 3 different responses. First that the parameter file was created but without any values read. Second it would read the default values, tell you what the values are and modify them. Third it will read the modified values print them out and tell you that it found modified values and not the defaults.

import os,struct
# We'll use os to test for the existence of the file, an struct to
# interact with the binary data.

FormatString='10sddi'
'''
The format string tells several parts of the program what we are working
with in regards to binary data. The format string listed above represents
a 10 character string, 2 double precision real numbers, and an integer.
Same as before but I added the int to show by example. Note that it is
just a string with letters representing different kinds of data.
'''

ParFileName='testbinary.par'
# The file by this name will show up in the directory the script is run in.

print 'Bytes occupied by binary data',struct.calcsize(FormatString),'\n\n'
'''
Sometimes you need to know how many bytes to read from a file in order
to get one set of data when multiple sets exist, a good example goes
back to the coordinate file mentioned earlier. In order to read and process
one point at a time you can only read the necessary bytes. the calcsize
method in struct will read a format string and tell how many bytes are
represented. You can go into a python interpreter, import struct then
evaluate some single item format strings like the following for a
float, a double, an int, and a 1 character string.
>>> struct.calcsize('f')
4
>>> struct.calcsize('d')
8
>>> struct.calcsize('i')
4
>>> struct.calcsize('s')
1
'''

if not os.path.isfile(ParFileName): # If the file doesn't exist.
...ParFile=open(ParFileName,'wb')
...# Then open it in write (create or erase) mode with a binary modifier

...BinaryData=struct.pack(FormatString,'Word',.2,.0,0)
...'''
...The pack method in the struct module uses the format string to process
...the data that comes next into the bytes which represent them in binary
...form. It is important that the amount and type of data following the
...format string match, notice I am passing in a string (less than 10
...characters allowed, 2 real numbers and an integer.
...'''

...ParFile.write(BinaryData)
...ParFile.close()
...# We have a binary file and binary data so just and write and close it.

...print 'Program run with no existing file'
...print 'Data set to defaults.'
...print 'testbinary.par should now exist.'
else:
...ParFile=open(ParFileName,'r+b')
...# If it does exist open for reading + writing with binary modifier.

...BinaryData=ParFile.read()
...'''
...This time since the file exists we are going to get the binary data
...right out of the file with a read. Since we want the whole file there
...is no reason to use calcsize to tell it how many bytes to read.
...'''

...DataList=struct.unpack(FormatString,BinaryData)
...# Going the other direction, lets use unpack to use the format string
...# to convert the binary data into a list of values.

...print 'Data from file, note it is a list\n ',DataList
...# and print the list.

...ParFile.seek(0,0)
...'''
...Now assuming that there are new values that need to be used to update
...the parameter file. The first thing we need to do is go back to the
...beginning of the file, read up on seek, but this basically moves you
...0 bytes from the beginning. If in the coordinate file example I had a
...data structure that require 24 bytes and wanted to get the 3rd value
...from the file I could to the 48th byte and read 24. This way you can
...read, write, and overwrite data in place within a file.
...'''

...print 'File position after seek ',ParFile.tell()
...# Every open file has a tell method which gives the current position.

...BinaryData=struct.pack(FormatString,'Updated',1,2,3)
...ParFile.write(BinaryData)
...# Same idea as above, just using the changed data (wherever it came from)

...if DataList[0].split('\x00')[0]=='Updated':
......print 'You have seen it all.'
......print 'Remove testbinary.par to start over'
......'''
......There may be a better way to convert this 10 character string to
......what you normally think of as a string, but it comes back padded
......with the 0 bytes and this works simply enough. Think of it this
......way 'Updated\x00\x00\x00' has 3 extra bytes, if you split on that
......character you get back ['Updated', '', '', ''] of which 'Updated'
......is the first or [0] element. Ugly but I've never really look into
......a better way.
......'''

...else:
......print 'Program run with an existing file'
......print 'Data was updated run again to see change'
......# 2nd run string doesn't equal 'Updated' so print appropriate message.

...ParFile.close()
...# Good form to explicitely close files.

I need to wrap it up for now, I'll have to save the parsing command for next time.

Life Outside Vr 4

We've looked at external files with the goal of getting useful information out of them. At the same time how about saving something there. We use a program that takes symbols in a certain layer, compares their elevation to an active dtm, and if they are over a specified distance from the elevation displays the error. It is a very useful program and when it was first written had all the variables hard coded, but stored at the beginning of the file so they could be easily found and changed. With the advent of PromBox it just became too easy to allow the user to confirm and change the variables each time. You could even easily use the old hard coded values as defaults. However it seemed like a certain set of values would be used over and over again based on job specifications. The obvious solution would be to have the last values used stored between runs to be used as the new defaults. Storing values like this is easy enough, but what elements would the most elegant solution contain? I have no idea but I did come up with a couple of features I really wanted.

First lets store them in a logical location like /vr/hostdir where all the other function parameter files are stored. Truthfully it would probably be better to make a user parameter directory, but I'll be lazy for now because the next thing I wanted was a test to see if the parameter file existed already and if not, would create one with the original default values. Yes I could test for a folder and if it wasn't there create it but maybe that will come in the next version but for now I didn't really want to change the directory structure.

import os
# os is necessary to test the existence of the parameter file.

ParFileName='c:/vr/hostdir/save_params.par'
# Call it what you want, this is great code to save for later use in other
# programs that require persistence.

if not os.path.isfile(ParFileName): # Does the file exist.
open(ParFileName,'w').write('%d %.3f %.3f'%(86,.2,0.0))
# If not then create it in write mode and write an integer and 2 floating
# point numbers with 3 decimal places.

Params=open(ParFileName,'r').read().split()
# Since it either existed or was created all we need to do now is open, read
# and split the values into a list that would look like ['86','.2','0.0']

PromBox=VrPromBox('Display Sym to Dtm',20,1)
PromBox.AddInt('Layer to Check',int(Params[0]),1,1000)
PromBox.AddDouble('Display delta >',float(Params[1]),2)
PromBox.AddDouble('Text Rotation',float(Params[2]),1)
# Using the values stored in the Params list make a prombox which will allow
# the user to change the values.

if (PromBox.Display(0) == 0):
...LayerToCheck=PromBox.GetInt(0)
...DisplayDelta=PromBox.GetDouble(1)
...TextRot=(PromBox.GetDouble(2))
# Grab the new values.

open(ParFileName,'w').write('%d %.3f %.3f'%(LayerToCheck,DisplayDelta,TextRot))
# And write them out to a file.

I won't spend the time on all the uses for keeping persistent variables between sessions, but this is the basics.

This just about wraps it up for reading and writing files, except for maybe writing parameters as a binary file, just for the fun of it and maybe walking through the single line parser I posted on the forum. It isn't very flexible but offers some interesting lessons. Here's a peek.

alist=[]
for line in open('hughes.pxyzopk','r').readlines():Alist.append(line.split())

Life Outside Vr 3 (well not very far outside)

Actually not at all, but the point isn't that we need to work outside Vr, it is that there is a whole world of files available using some straight python functionality. Now that we have a little file opening and reading under our belts, lets come back into Vr and bring a little of that technique with us. Let's also work on tightening up the code a little. If you are fairly new to VrPython, I think you'll really like this one. We are going to let Vr tell us the name of the file to open (in this case the current function key file) then using standard python we'll open, read, and parse the data we want out of the file, then use Vr to build a prompt box. For this example the data we want is all the function key names, and the prompt box will allow us to pick a function key name. Why? Well how about if you were going to write a parallel function using the existing inspar command but wanted to change the PARMOD= to "Function Key" and wanted to select the function key from a list then send it using the FKEY= key-in. If so, here is one way to get the function key name. It is rather long and convoluted but I'll try to make it worth what you are paying for the look.

Cfig=VrCfg()
# We'll need a Vr config object to access the name of the currently
# loaded function key file name.
FileName=Cfig.GetFkeyFileName()
# Grab the name.
FkeyFile=open(FileName,'r')
# Open the file
FileData=FkeyFile.read()
# We're going to read the file twice. This time as a string so we can count the
# instances of the key-word "KeyName" to figure out how many entries our
# prompt box will need.
NumberOfFunctionKeys=FileData.count('Fkey KeyName')
# Use the string method count to do just that.
PromBox = VrPromBox ("Function Keys", 60, 1)
# Create a PromBox object.
PromBox.AddList ("Select a Function Key",NumberOfFunctionKeys , 40, 0)
# I'll skip helpful text and just get right to the listbox of length equal to
# the number of function keys.
FkeyFile.seek(0,0)
# Rewind the file to read it again.
FileData=FkeyFile.readlines()
# This time read it as a list, each element of which is a line from the file.
for FileLine in FileData:
# For each line in the data list.
..if FileLine.count('Fkey KeyName') != 0:
# Test to see if it is a line containing the function key name.
....FkeyName=FileLine.split()[2]
# If so split the line data into the three words
# 'Fkey','KeyName','ActualName' which is in itself a list, the third element
# of which ( [2] ) is the actual name of the function key.
....PromBox.AddListItem(FkeyName)
# Add that to the list of items to be displayed.
FkeyFile.close()
# Could have closed it after the readlines() but just go ahead and do it now.
if (PromBox.Display(0) == 0):
..FunctionKey=PromBox.GetList (0)
# Standard prompt box usage to grab a list item
print FunctionKey
# and print it out (or use it in an interesting way)

Now as far as tightening it up. I could have read the file only once and built a for loop to count the instances of the KeyName key-word, but I'm going to go in a totally different direction. I'm still going to read the file twice, as a matter of fact I'm going to open it twice. The goal of this exercise is to get as few lines of code as I can. I'm not going to profess that this is the tightest and most efficient. It just has a lot less lines of code and is intended to make you think. Look at it first, then read the explanation.

PromBox = VrPromBox ("Function Keys", 60, 1)
PromBox.AddList ("Select a Function Key", open(VrCfg().GetFkeyFileName (),'r').read().count('Fkey KeyName'), 40, 0)
for FileLine in open(VrCfg().GetFkeyFileName (),'r').readlines():
..if FileLine.count('Fkey KeyName'):PromBox.AddListItem(FileLine.split()[2])
if (PromBox.Display(0) == 0):print PromBox.GetList (0)


Line 1: We have to create a prompt box object because I can't find any way around it and we'll need to use methods of this specific object in the next line.

Line 2: Let's take this a step at a time.
A) Add to the list using the formula AddList (Prompt, NumItems, Height, DefaultItem) Prompt is set to "Select a Function Key", NumItems is the tricky part, Height is 40 and DefaultItem is 0.
B) Look at the NumItems formula and read it as follows
Open a file using the string returned by a VrCfig() object method GetFkeyFileName in read mode ( 'r' ) which open file object you should read() the data, of which data you should count() the number of occurrences of 'Fkey KeyName'. Notice that when all this is said and done it just give a number which represents the number of function key names and therefore the number of function keys

Now that we have a prompt box which contains a list set to the length equal to the number of function key names, let's populate the list with those names.

Line 3: For each element in a list, the list comes from opening the same file again, in read mode again, but this time instead of read() to get one solid string, we use a method that will return a list which is exactly what the for loop is working on. We are looping on a list that was created and populated just for the loop, each element of which is a line from that file.

Line 4: FileLine represents the line of data out of the file which is being examined as we loop over the entire list of lines. count() tells how many occurrences of a work exist in the string 0 means none which in terms of an if means false so any other number that comes back means true (the string exists, therefore it is a function key name). if it is true we want to add the actual name to the list. The actual name as we saw above means splitting the string, which creates a list, from which we want element [2]. So add to the list the third word which is the result of splitting the string because it does (true) contain the phrase 'Fkey KeyName'. I didn't add another line and indent it because there is only one statement to execute if true which allows you to just put it after the colon and be done with it.

Line 5: Same idea, standard prombox syntax, but since there is only one action, just put it on the same line. Ugly code but remember my only goal was to reduce the line count. And why put the string in a variable just to print it on the next line, print it as it returns from GetList()

And I haven't even taken the time to explain the forum post regarding reading and parsing aerosys .pxyzopk files, a great example of the kind of data we mappers work with all the time. Maybe next time.

Life Outside Vr 2

Ok I have a few minutes so let's look at actually reading some data out of an open file. We'll stick with ascii files for now. There are so many fun reasons to work with files outside Vr that it would take too long to enumerate all of them but a few of my favorites include.

• Reading info out of an ImageUtil project and creating all the Vr2Ori project and all models with 1 click.
• Converting all Vr2Ori and VrAt files to or from their corresponding equivalents in orientp3 (yes we still use it and get along fine) so a setup in the analytical equals a setup in softcopy or vice versa.
• Reading a PATB and plotting it graphically in VrOne for visual analysis and blunder checking.
• Reading and plotting AT results including photos, layout data, computed extents and photo scale based on image centers and control averages.

You get the idea, there are just so many files out there that can be of easy use with a click or two. Sometimes there is already a way to do the same things, but a custom program allows you to set things up exactly the way you want without any further intervention. As in the last post we'll be working with vr.cfg in read only mode because it is a file that exists on everyones system (if you are working in Vr, which I assume) and has the same basic patterns. Again I will suggest you work in some kind of python IDE, I suggest pythonwin but you could even work from the python command interpreter if you want.

Let's start by creating a python file object like we did last time, it will be called f (for file).
f=open('/vr/hostdir/vr.cfg','r') # 'r' means read mode.
now if I
print f
there should be some obscure text describing the object. Hopefully there was some time to look over the file object docs to study up on the simple methods, let's look at a few.
Read one line out of the open file
>>> f.readline()
'# VR Configuration File\n'
I should note here that once it reads the line, the address in the file that python is sitting at is the beginning of the next line so if you do it again
>>> f.readline()
'# FileName : c:\\vr\\hostdir\\vr.cfg\n'
if for some reason it is necessary to go back to the beginning of the file we need to "seek" the new position. seek takes two arguments "offset" (how much to jump from the position passed in) and "whence" (where to start the offset). "Whence" defaults to the beginning of the file, so if you want to go the the beginning you can say "see" with an offset of 0 (don't move at all from the position passed in) and a whence of 0 (or if you pass nothing for whence this is the default) so
f.seek(0,0) # 0 offset, 0 whence
f.seek(0) # 0 offset, default whence which is 0
will both take python to the beginning of the file, so do that then readline() again and you should see the first line again. Let's read that first line and store it as a string we'll call "s"
>>> s=f.readline()
>>> print s
# VR Configuration File
It would of course be possible to do this with each line of the file, though there are better methods to be discussed later, but lets look a what we could do with that string now that we have it.
I can split it based on white spaces (the default for split) which returns a list of the words in the string.
>>> s.split()
['#', 'VR', 'Configuration', 'File']
For that matter we could split it based on any character like commas, or in this case the "i"
>>> s.split('i')
['# VR Conf', 'gurat', 'on F', 'le\n']
Note that splitting on white spaces takes off the newline character but splitting on a printable character doesn't. Of course there are plenty of ways to take the newline off, but just take note for now. How about if we needed to test for a sub string inside the string, well let's count occurrences.
>>> s.count('i')
3
>>> s.count('ig')
1
>>> s.count('z')
0
So a test would look something like
>>> if s.count('ig') : print 'yes'
...
yes
I didn't say if s.count equals something or is greater than something because I just wanted to know if it was non zero and anything non zero is true for the if test.

There is no sense typing out all the things you can do with strings because they are documented in the python docs, it is worth noting that there used to be a string module, but now each of these methods is a built-in which is why there was never an import statement above.

Let's go back to the split step and put the resulting list in a variable called "l", then we can loop over the list and do whatever we want, in this example just print each item.
>>> l=s.split()
>>> l
['#', 'VR', 'Configuration', 'File']
>>> for item in l:
... print item
...
#
VR
Configuration
File
or grab individual elements of the list.
>>> l[3]
'File'
or test to see if the list contains certain values.
>>> l.count('File')
1
>>> l.count('Nothing')
0
Like strings I'll leave the explanation of lists and their methods to the professionals, in this case the python tutorials.
But I'll leave with a couple of thoughts:
First, check out readlines() it will come in handy and will probably bear heavily in the next post.
Second, one of the real beauties of python is it's ability to get a lot done with a little code, this can be accomplished by taking whatever data a method returns and acting on it with the appropriate method. Consider the following lines typed into the interpreter.

open() is going to return a file object with all it's accompanying methods.
>>> open('/vr/hostdir/vr.cfg','r')
obscure text describing an open file object

let's just tack the readline() onto the end and let it return the first line which is a string.
>>> open('/vr/hostdir/vr.cfg','r').readline()
'# VR Configuration File\n'

It it is a string we can split it to get a list of words that were included.
>>> open('/vr/hostdir/vr.cfg','r').readline().split()
['#', 'VR', 'Configuration', 'File']

Of course the list is made up of elements that can be addressed lets get the second element whose index is 1.
>>> open('/vr/hostdir/vr.cfg','r').readline().split()[1]
'VR'

Which is a string and can be forced to lower case using the lower() method.
>>> open('/vr/hostdir/vr.cfg','r').readline().split()[1].lower()
'vr'

Is this stuff cool or what.

Life outside Vr

I have had several folks ask about opening reading and parsing external files. There are so many applications that I don't even know where to begin. Whether plain ascii text files, or binary, python is loaded with tools for doing what you want or need. I think I'll just start rambling and see where we head. One thing that you will notice immediately is that there can be literally dozens of ways of accomplishing the same task. I'm not going to pretend that what I am putting forward is the best or most efficient, I'll just mention some tools that come in handy quite often. For today lets just open a file and grab some data from it. Let's pick a file that everyone should have, that being your Vr configuration file \vr\hostdir\vr.cfg. Don't worry, for now we'll just open it in read only mode. In your favorite python editor or pyedi within VrOne type and run the following.

f=open('\vr\hostdir\vr.cfg','r')

Ok, I'll admit that was a trick, it shouldn't have worked, even though everything about the command is totally correct, except that is for the backslashes "\". But wait you say, we are working in windows, and the backslash is what is always used to specify path hierarchy, and you would be correct. The problem is that the backslash is a special character in python (and several other languages, not to mention unix). Python uses a forward slash to separate path elements, or to use backslashes you have to tell python you are sending it a "raw" string and that it should not treat the backslashes as escape characters. I don't start here to be bothersome, but it is important to understand that python expects certain things and may send back something you aren't expecting if you aren't careful. Note the paste of a session where I open the file using 3 valid commands, look at what comes back, and realize that this is outside VrOne in the PythonWin editor.

First I open it with forward slashes because I grew up on unix, the forward slash is always in the same place no matter what keyboard you have, it is what python prefers, and I just like it better. I'll use "f" as the open file object and will open it, display it, and close it. The 'r' means to open it in read-only mode, we'll talk about modes later but I digress.

>>> f=open('/vr/hostdir/vr.cfg','r')
>>> f
Text describing file object which blogger deletes for some reason.
>>> f.close()

Next we'll use backslashes, but they have to be protected by another character so they aren't evaluated as an escape sequence, by coincidence the character that does this is a backslash so each occurence must be entered as
>>> f=open('\\vr\\hostdir\\vr.cfg','r')
>>> f
Text describing file object which blogger deletes for some reason.
>>> f.close()

Next we'll use the backslashes but we'll tell python that the name string is a raw string and that the backslashes shouldn't be evaluated as anything else. Just put an r in front of it and we're good to go.
>>> f=open(r'\vr\hostdir\vr.cfg','r')
>>> f
Text describing file object which blogger deletes for some reason.
>>> f.close()

It is worth noting that if you grab a filename inside Vr using either of the following, it returns forward slashes.
print PyVrGui().OpenFileNameDialog('title','c:/vr/hostdir','*.cfg')
C:/vr/hostdir/vr.cfg
print PyVrGui().OpenFileNameDialog('title','c:\\vr\\hostdir','*.cfg')
C:/vr/hostdir/vr.cfg

That will do for now. Yeah it's pretty slim, but I need to go and we'll actually read something next time. For now though it wouldn't hurt to get familiar with some of the python file methods, from the python docs. You can also get a little more information on the open function from the built-in functions docs (just scroll down to open()), this will come in handy when we move to appending, writing, and binary files.

Utility programs 3

Ok, time flies and all, but we better get some actual working code in this thing. In post 1 all we did was create a usage string and test argv to see if anything had been passed in (in this case we don't want anything). In the second we used glob to build a list of files matching a certain extension in the directory we're working in. Eventually we could us a windows file select dialog but for now this will be a simple command line utility. Let's get right to the pertinent section then put it all together in the end. Using th same indentation as where we left off, lets replace

........print os.path.splitext(File)

which was just used to show that something was working and to illustrate os.path functions with

# Step through the list of files processing each
....for File in Filelist:
# Each file will need to be opened
........OppFile=open(File,'r')
# readlines() reads every line in a file, returns a list in which each member
# of the list is a line in the file, including the end of line character.
........FileData=OppFile.readlines()
# Step through the list of lines in the open file
........for Line in FileData:
# This isn't real tricky or elegant, but it turns out that the coordinate
# positions stored in the .opp file are on lines which have an "l" as the
# second character, so let's just determine first of all if the data line
# is a coordinate
............if Line[1]=='l':
# Then if it is, the first character determines which coordinate it is.
# If it's an X, Y, or Z we'll use the split(function) to break the line
# into a list of strings, which contains the number as the second
# element, then use float() to convert it to a real number, and store it
# in the appropriate variable.
................if Line[0]=='X':
....................X=float(Line.split()[1])
................elif Line[0]=='Y':
....................Y=float(Line.split()[1])
................elif Line[0]=='Z':
....................Z=float(Line.split()[1])
# Once we have parsed each line in the file to pull out the coordinates
# lets use some formatted printing to output the file name and values.
# Remember that "File" contains the name of the file with the extension
# so the split using the dot will give a list ['filename','extension']
# so the base name is element 1 or [0]
# Close the file and your on your way.
........print '%s %.3lf %.3lf %.3lf'%(File.split('.')[0],X,Y,Z)
........OppFile.close()

so in the end after removing the modules that were only temporary, the final program looks like

import sys,glob

def usage():
....'''
....Explanation text to print if anything is entered as an argument.
....'''
....print '\nUsage: python opp2txt.py\nThere are no arguments, it will just look in the current directory\n for files with a .opp extension, strip the coordinates out of them\n and output them to the screen\n\nListings can be redirected into a file.\n eg: python opp2txt.py > photos.txt'
....sys.exit(1)

if len(sys.argv)==1:
....Filelist=glob.glob('*.opp')
....if Filelist==[]:
........print "There don't appear to be any .opp files in this directory"
........sys.exit(2)
....for File in Filelist:
........OppFile=open(File,'r')
........FileData=OppFile.readlines()
........for Line in FileData:
............if Line[1]=='l':
................if Line[0]=='X':
....................X=float(Line.split()[1])
................elif Line[0]=='Y':
....................Y=float(Line.split()[1])
................elif Line[0]=='Z':
....................Z=float(Line.split()[1])
........print '%s %.3lf %.3lf %.3lf'%(File.split('.')[0],X,Y,Z)
........OppFile.close()
else: usage()

Now this just prints to the screen and would have to be redirected to a text file. For the fun of it the first modification once we know it is running would be to send it directly to a file. As always, final working code is found on my personal VrPython website.

Utility programs 2

Rather than retyping the code from the previous post, I'm just going back to the line with argv and proceed from there with some actual working code. This time I want to focus on getting the filenames that will be processed. Remember the test of argv was just to determine if any arguments were passed. Since this program doesn't take any, having something else appear on the command line indicates a lack of understanding so we printed some usage instructions. Now let's assume the user knows that if he types "python opp2txt.py" he will get every opp file in the directory processed. But how to build a list of file names? Today we'll just build the list and print the filenames to show that it worked. The magic we want is contained in the glob module, no I don't know why it is named that, but isn't that just about the coolest name around? Here is the code that could be inserted into opp2txt.py to grab filenames.

if len(sys.argv)==1:
....Filelist=glob.glob('*.opp')
....if Filelist==[]:
........print "There don't appear to be any .opp files in this directory"
........sys.exit(2)
....for File in Filelist:
........print os.path.splitext(File)
else: usage()


First let's look at glob. The module itself is fairly simple and can be found in the python module docs. The short story is that you can return filenames matching a pattern in a specified directory into a list of strings. Really the best way to illustrate is to try it. In your favorite python editor do the following.

import glob
glob.glob(r'\vr\hostdir\*.par')

This should return a list which contains the names of every .par file in that directory. Something like...
'\\vr\\hostdir\\drivefile.par', '\\vr\\hostdir\\editline.par',
The lower case r before the path name just tells python to treat the string that follows as a raw string and not to see the backslashes as special characters. You could also put in forward slashes without the 'r' since that is a valid path separator in python but the return list comes back as an odd mix of forward and backward slashes.

We are storing the returned list in a variable called Filelist. If there are no .opp files in the directory the list will be empty in which case we print a message to that effect and exit, I just picked a random exit code of 2 to indicate no files found.

Next the for loop will step through the list and process each name string in the list. The variable which represents the name string will be "File".

Meaningful processing will come next time, but for now we could just use the code "print File" to display each filename but let's take this opportunity to explore the extremely useful os.path functions. These are most helpful in separating components of full filenames into paths, extensions, base filenames, etc. In the program which will be run in a directory that includes the .opp files and will not contain a path component the list would look something like...
['1.opp', '2.opp']
and the splitext() will just return a name and an extension similar to...
('1', '.opp')
('2', '.opp')
which could be accessed individually using indexes
>>> print ('2', '.opp')[0]
2
>>> print ('2', '.opp')[1]
.opp
>>>
However to really have some fun lets go back to the previous example in the editor and try some examples with the name list returned from the vr hostdir directory. Here are some truncated examples of what would be returned. Take time to play with the os.path functions, there is some real usefulness there.

>>> for name in glob.glob(r'\vr\hostdir\*.par'):
... print os.path.basename(name)
...
drivefile.par
editline.par

or if we run it through splitext() after it has been stripped to basename.

>>> for name in glob.glob(r'\vr\hostdir\*.par'):
... print os.path.splitext(os.path.basename(name))
...
('drivefile', '.par')
('editline', '.par')

or how about just running splitext to get the filename so that you can create another file with the same name, but with a different extension.

>>> for name in glob.glob(r'\vr\hostdir\*.par'):
... print os.path.splitext(name)
...
('\\vr\\hostdir\\drivefile', '.par')
('\\vr\\hostdir\\editline', '.par')

or what about if you want to get the path name so you can create a file in the same directory.

>>> for name in glob.glob(r'\vr\hostdir\*.par'):
... print os.path.split(name)
...
('\\vr\\hostdir', 'drivefile.par')
('\\vr\\hostdir', 'editline.par')

or just test whether or not the file exists, of course all these do because we're just passing in existing file names, but you get the idea.

>>> for name in glob.glob(r'\vr\hostdir\*.par'):
... print os.path.exists(name)
...
True
True

Cool stuff, and we haven't even done anything useful yet, hopefully next time.