Python file exist

Just provide a quick reference to myself. I forget the way of checking file exist from time to time.

>>> import os
>>> os.path.exists('c:\\abc\\file.txt')
True
>>>


>>> import os
>>> os.path.isfile('c:\\abc\\file.txt')
True
>>>

Create Spatialite database in QGis

Create Spatialite DB in QGis with Python is quite easy.

http://www.gaia-gis.it/spatialite-2.4.0-4/spatialite-cookbook/html/python.html has a very good tutorial.

Some remarks:
1. Initiality I use QGis 1.6 Standalone version, the spatilite version is 2.3.x. We need to add srid info into spatial_ref_sys manually.

2. Sample codes in QGis 1.6 plugin:

from pyspatialite import dbapi2 as sqlite3
dbName = str(self.txtOutputDB.text()) #textOutputDB is a lineEdit object(PyQt)

#if os.path.exists(dbName) :
ocon = sqlite3.connect(dbName)
ocur = ocon.cursor()

sql = 'SELECT InitSpatialMetadata()'
ocur.execute(sql)
#add projection information
sql = "INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, ref_sys_name, proj4text) VALUES (4326, 'epsg', 4326, 'WGS 84', '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs');"
ocur.execute(sql)

You can get the INSERT code from the following link:
http://www.gaia-gis.it/spatialite/init_spatialite-2.3.zip

3. After I switch to QGis 1.7, the spatialite version is updated, the projection info will be automatically created. However, an empty Spatialite database will have 3MB in size as there are many projection information in spatial_ref_sys table.

4. To reduce the size, I just keep several rows and delete unwanted srid in spatial_ref_sys table:

#delete unnecessary spatial_ref_sys
sql = 'DELETE FROM "spatial_ref_sys" where srid!= 2326 and srid!= 4326'
ocur.execute(sql)
sql = 'VACUUM'
ocur.execute(sql)

Python strtok equvialent

C programmers usually use strtok to perform operation on string with standing patterns. In Python, we can use re.split to get similar result.

Example:

>>> import re
>>> st = "9394 811 813 815"
>>> result = [str(x) for x in filter(None, re.split('[ ,\n,\t]',st))]
>>> print result
['9394', '811', '813', '815']
>>>

Python read binary file

To read binary file,

fp = open(ifile,"rb")

To read 1 byte from fp,

byte = fp.read(1)

To read 30 bytes from fp,

block = fp.read(30)

block = fp.read(30)

To read the bytes from a block of binary data,

(Year,) = struct.unpack('b',block[3:4])
(Month,) = struct.unpack('b',block[4:5])
(Day,) = struct.unpack('b',block[5:6])
(Hour,) = struct.unpack('b',block[6:7])
(Minute,) = struct.unpack('b',block[7:8])
(Second,) = struct.unpack('b',block[8:9])
recdate = str(Year) + '-' + str(Month) + '-' + str(Day) + ' ' + str(Hour) + ':' + str(Minute)

You should include "import struct" so that you can use the struct module.

To read text from binary data:

(Cellname,) = struct.unpack('8s',blk[2:10])

The format characters are repeated here for quick reference.

Format	C Type	Python type	Standard size	Notes
x	pad byte	no value
c	char	string of length 1	1
b	signed char	integer	1	(3)
B	unsigned char	integer	1	(3)
?	_Bool	bool	1	(1)
h	short	integer	2	(3)
H	unsigned short	integer	2	(3)
i	int	integer	4	(3)
I	unsigned int	integer	4	(3)
l	long	integer	4	(3)
L	unsigned long	integer	4	(3)
q	long long	integer	8	(2), (3)
Q	unsigned long long	integer	8	(2), (3)
f	float	float	4	(4)
d	double	float	8	(4)
s	char[]	string
p	char[]	string
P	void *	integer		(5), (3)

More info of using struct can be found here:
http://docs.python.org/library/struct.html

Python Bitwise Operators:

Bitwise operator works on bits and perform bit by bit operation.
Assume if a = 60; and b = 13; Now in binary format they will be as follows:
a = 0011 1100
b = 0000 1101
-----------------
a&b = 0000 1100

a|b = 0011 1101
a^b = 0011 0001
~a  = 1100 0011
There are following Bitwise operators supported by Python language

Operator	Description	Example
&	Binary AND Operator copies a bit to the result if it exists in both operands.	(a & b) will give 12 which is 0000 1100
|	Binary OR Operator copies a bit if it exists in eather operand.	(a | b) will give 61 which is 0011 1101
^	Binary XOR Operator copies the bit if it is set in one operand but not both.	(a ^ b) will give 49 which is 0011 0001
~	Binary Ones Complement Operator is unary and has the efect of 'flipping' bits.	(~a ) will give -60 which is 1100 0011
<<	Binary Left Shift Operator. The left operands value is moved left by the number of bits specified by the right operand.	a << 2 will give 240 which is 1111 0000
>>	Binary Right Shift Operator. The left operands value is moved right by the number of bits specified by the right operand.	a >> 2 will give 15 which is 0000 1111

Install 3rd party Python packages into QGIS

When you write plug-in for QGIS, you may need to rely on additional python packages to work with. However, I find no clue from qgis.org and forum search. After trial and error, here is the approach I used to install 3rd party python packages to QGIS.

Configuration

In my Windows XP, I have 2 copy of Python 2.5.2. One is the original stock version downloaded from www.python.org and installed at c:\python25. Another of course is the one come with QGIS. If you issue

import sys
sys.path

in stock Python console and QGIS Python console respectively, you will get two different results:

Result from stock Python console:

>>> import sys
>>> sys.path
['', 'C:\\windows\\system32\\python25.zip', 'c:\\python25\\DLLs', 'c:\\python25\\lib', 'c:\\python25
\\lib\\plat-win', 'c:\\python25\\lib\\lib-tk', 'c:\\python25', 'c:\\python25\\lib\\site-packages']

Result from QGIS Python Console:

>>> import sys
>>> sys.path
['C:/PROGRA~1/QUANTU~1/apps/qgis/./python', 'C:/Documents and Settings/19/.qgis/python', 'C:/Documents and Settings/19/.qgis/python/plugins', 'C:/PROGRA~1/QUANTU~1/apps/qgis/./python/plugins', 'C:\\PROGRA~1\\QUANTU~1\\apps\\gdal-17\\pymod', 'C:\\Program Files\\Quantum GIS Copiapo', 'C:\\PROGRA~1\\QUANTU~1\\bin\\python25.zip', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\DLLs', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\plat-win', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\lib-tk', 'C:\\PROGRA~1\\QUANTU~1\\apps\\qgis\\bin', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\site-packages', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\site-packages\\win32', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\site-packages\\win32\\lib', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\site-packages\\Pythonwin', 'C:\\PROGRA~1\\QUANTU~1\\apps\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode', 'C:\\PROGRA~1\\QUANTU~1\\apps\\qgis\\python\\plugins\\fTools\\tools']

The above info tell us that Python package install in c:\python25 cannot be shared by Python in QGIS.

easy_install

If you want to install easy_install into QGIS, the setuptool will just tell you that it can only find one copy of Python in c:\python25 in registry:

To let setuptools find QGIS's python copy, go to Registry and look for HKEY_LOCAL_MACHINE\SOFTWARE\Python and export the selected branch to python25.reg beofore any modification. The exported python25.reg will look like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Python]

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore]

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5]

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\Help]

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\Help\Main Python Documentation]
@="C:\\Python25\\Doc\\Python25.chm"

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\InstallPath]
@="C:\\Python25\\"

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\InstallPath\InstallGroup]
@="Python 2.5"

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\Modules]

[HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.5\PythonPath]
@="C:\\Python25\\Lib;C:\\Python25\\DLLs;C:\\Python25\\Lib\\lib-tk"

In the registry, there is only information of the Python installed in c:\python25 folder. Select HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonPath and change the path to QGIS one. That is:

replace c:\python25 to C:\Program Files\Quantum GIS Copiapo\apps\Python25

After change, export the Python branch again and save it as qgispython.reg for future use.

Also, you also need to add C:\Program Files\Quantum GIS Copiapo\bin and C:\Program Files\Quantum GIS Copiapo\apps\Python25 into PATH. I removed c:\python25 from the PATH when I add the above items into PATH. After installation, I revert the change by adding back c:\python25 and remove the two items.

If you haven't install MinGW, install it and at c:\mingw\bin into PATH. You may also need to tweak the cgwincompiler. Check here for detail.

After the above setups, setuptools should be able to install in QGIS folder:

However, if go to C:\Program Files\Quantum GIS Copiapo\apps\Python25\Scripts and execute easy_install, you will get no responds. The program just return nothing. In this case, you can use easy_install-script.py to install packages. For example, to install Jinja package,

C:\Program Files\Quantum GIS Copiapo\apps\Python25\Scripts>python easy_install-2.5-script.py pyodbc
Traceback (most recent call last):
File "easy_install-2.5-script.py", line 5, in
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

C:\Program Files\Quantum GIS Copiapo\apps\Python25\Scripts>python eas_install-script.py jinja
python: can't open file 'eas_install-script.py': [Errno 2] No such file or directory

C:\Program Files\Quantum GIS Copiapo\apps\Python25\Scripts>python easy_install-script.py jinja
Searching for jinja
Reading http://pypi.python.org/simple/jinja/
Reading http://wsgiarea.pocoo.org/jinja/
Reading http://jinja.pocoo.org/
Best match: Jinja 1.2
Downloading http://pypi.python.org/packages/2.5/J/Jinja/Jinja-1.2-py2.5-win32.egg#md5=27b0804a126c2c
0ebd4a9dacffb9dcbb
Processing Jinja-1.2-py2.5-win32.egg
creating c:\progra~1\quantu~1\apps\python25\lib\site-packages\Jinja-1.2-py2.5-win32.egg
Extracting Jinja-1.2-py2.5-win32.egg to c:\progra~1\quantu~1\apps\python25\lib\site-packages
Adding Jinja 1.2 to easy-install.pth file

Installed c:\progra~1\quantu~1\apps\python25\lib\site-packages\jinja-1.2-py2.5-win32.egg
Processing dependencies for jinja
Finished processing dependencies for jinja

If you can issue "import jinja" in QGIS Python console without error, it means the installation is done.

Hope the above can help.

p.s. after install the 3rd party package, I will revert the registry (by double click python25.reg saved before )and edit the PATH (add c:\python25 and remove QGIS entries)

Recent search find another approach:
http://forum.qgis.org/viewtopic.php?f=2&t=8201

Use of Python in TEMS Investigation

TEMS Investigation is widely used in celluar network operators to carry out drive test activities. It supports Python to decode TEMS logfiles to user specific format.

Python Environment Setup

Currently, I am using TEMS Investigation 11.0.5. If you follow the manual of TEMS Invesitgaton to setup Python environment (PYTHONPATH for TI and PATH for Python25), you may still hit the following error:

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'TEMS.DataModel, Version=11.0.0.0, Culture=neutral, PublicKeyTok
en=148637998f549f7b' or one of its dependencies. The system cannot find the file specified.
File name: 'TEMS.DataModel, Version=11.0.0.0, Culture=neutral, PublicKeyToken=148637998f549f7b'

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

It seems there is a bug with the TEMS.pyd in the Application folder of TI. Copy the python.exe in c:\python25 to C:\Program Files\Ascom\TEMS Products\TEMS Investigation 11.0\Application should be able to solve the problem.

Also, make sure your Python version is 2.5x. Manual of TEMS Investigation clearly stated:

Please note that precisely this version (2.5.x) is required, not any earlier or later version

Reference

Useful links and reference that I visited frequently.

QGIS

QGIS
QGIS API
OSGeo.org QGIS forum
QGIS forum
PyQGIS Developer Cookbook

Python

Python
Python Packages
pysqlite usage guide
Python Online Doc

Spatialite

Spatialite
Spatialite tutorial

PyQt

Qt Online Reference Documentation

Compile Python 2.5 Packages with MinGW

In Python 2.5, default compiler is Visual Studio 2003. If you don't have the complier, you may end up with the following error message:

error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I don't have Visual Studio 2003 and therefore I try to follow the msg to download MingW32. Here are the steps:

1. Download MinGW from http://sourceforge.net/projects/mingw/ and install. It seems we better install the complier in default folder location c:\MinGW.

2. During the installation, I selected "C++ Compiler", "MSYS Basic System" and "MinGW Developer ToolKit".



3. Update your PATH: Add the C:\MinGW\bin directory to the system PATH. You may need to log-off and log-in again to reflect the added path.

4. Go to C:\Python25\Lib\distutils, create a file distutils.cfg and add the following lines:

[build]
compiler=mingw32

5. If you compile the package at this stage with "Python setup.py install", you may get the following error:

File "C:\Python25\lib\distutils\cygwinccompiler.py", line 84, in __init__
get_versions()
File "C:\Python25\lib\distutils\cygwinccompiler.py", line 424, in get_versions

ld_version = StrictVersion(result.group(1))
File "C:\Python25\lib\distutils\version.py", line 40, in __init__
self.parse(vstring)
File "C:\Python25\lib\distutils\version.py", line 107, in parse
raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.20.51.20100613'

6. It says that your compiler version is invalid. Go to C:\Python25\Lib\distutils and backup a copy of cygwinccompiler.py. Open cygwinccompiler.py in text editor (e.g. notepad or IDLE). In def get_versions(): section you will find the following line:

result = re.search('(\d+\.\d+(\.\d+)*)',out_string)

Replace all "result = re.search('(\d+\.\d+(\.\d+)*)',out_string)" with the following code:

result = re.search('(\d+(\.\d+)?(\.\d+)?)',out_string)

It will tell Python to check the first digit only.

7. Try to compile again and it should work now.

8. Since I am working with Python 2.5.2, I am not sure 2.6 or 2.7 have that kind of issue or not.

Please tell me if it works for you are not.

Reference:
http://bugs.python.org/issue2234

About this blog

Welcome to Python code blog. Most likely, I think you reach this blog by searching info related to QGIS or Python. I hope you will find the blog useful.

Background

This is a blog about programming in Python. Or I should say it is about the problems that I encountered in creating QGIS Plugin with Python script. I am working in a mobile network company as an RF Engineer. My team started a project to port an old in-house GIS program written in Borland C++ to QGIS. As a telecom engineer, I try to use my limited programming skill to start the project.

In general, radio network planning team will have coverage planning tools such as Mentum and Atoll to assist their planning activities. The tools are quite expensive and license controlled. If we just want to display site information on GIS platform, Mapinfo is usually used. Mapinfo info is good and easy to use, but it is not free as well. Therefore, we try to find a suitable open-source platform to replace our old GIS tools.

Googling the web, we found several suitable GIS tools : MapWindow, Udig and QGIS. While all 3 GIS apps support programming or plug-ins, they are walking in different paths. MapWindow goes in dot net approach. If you are familiar with dot net and ActiveX, MapWindow may suit your need. Or if you are a Java programmer, you should consider Udig seriously. Since I am not good at programming, I choose an easier path of Python scripting approach.

After kick-off, I am almost submerged in the sea of QGIS, Python, PyQt, GIS file format and of course debugging process. It is quite difficult to remember everything without write it down clearly, especially to a "part-time programmer" like me. And I think it would be good to share the info to someone who needs to write QGIS plug-in in Python.

Without formal training or programming background, most of the codes and info shared are learnt by googling the web and wild-guess. So please accept my ignorance and your comments will be highly appreciated.

So you may ask what is the function of plug-in you are going to write? Screen captures are attached to give you some ideas:



Display of site inform on a map




Display signal level of drive test result


MP