Pattern of Error Reporting in Python

Instead of Intro

That is not an uncommon and even not rare when an application which was supposed to be a prototype suddenly becomes a tool for everyday usage. Not a perfect thing here is that in a rush to make the program more or less user-friendly a developer has to hide internals by preserving error messages with internal details. Here I am going to share the pattern I mostly use in my "so-called-prototype" Python scripts; especially when they are translated to binaries using py2exe.

The pattern allows to introduce an extended error reporting in Python scripts w/o any extra costs.

Own Errors

The only rule I follow when there is a need to raise an error is to forget about standard ready-to-use Python exceptions. Due the following reasons:

• Need to distinguish where an error came from: from the "Batteries" or from the application's logic;
• Need to express an application's domain in the code.

The rule is true even for a spaghetti-style code which is supposed to be thrown away tomorrow or even today; this will cost nothing but might help with debugging.

So introduce own exception class:

class Error(Exception):
    def __init__(self, message, innerError = None):
        msg = message
        if innerError:
            msg += " *- {0}".format(str(innerError))

        Exception.__init__(self, msg)

The exception class here is straight forward for the sake of simplicity. In serious applications it is much better to introduce a field for an inner error, environment etc.

Respect Each Error

When a script's function pass flow control to another one there is a bell that the flow goes to another virtual layer. Each new layer is worth to have own logged error if there is any.

Suppose below that foo() is a first layer, bar() is a second one. So the application might look like:

def tryRussianRoulette():
    ### NOTE: that is a very bad practice to put import statements somewhere in a logic
    import random
    
    isFired = (0 == random.randint(0, 5))
    if isFired:
        raise Error("bang!")
        
def bar():
    try:
        tryRussianRoulette()
    except Error, e:
        raise Error("Russian Rouletter has fired", e)

def foo():
    try:
        bar()
    except Error, e:
        raise Error("Failed to bar-bar", e)

def main():
    foo()

def propagateToUser(error):
    if isinstance(error, Error):
        print "[Error]", str(e)
    else:
        print "[Unknown error]", str(e)

if "__main__" == __name__:   
    try:
        main()
    except Exception, e:
        propagateToUser(e)
        sys.exit(1)

    sys.exit(0)

tryRussianRoulette() is a function which may cause an error.
When application is accidentally in a production, an error for end-user (!) would look like:

[Error] Failed to bar-bar *- Russian Rouletter has fired *- bang!

In most cases (if you have not skipped error handling on each layer), the error is descriptive enough to understand the problem.

Pattern in Action

How to introduce an ability for an extended error tracing w/o writing tons of extra code? The answer is to vary try/except statement's behavior depending on system's environment variable bound to an application.

The application's code above is just extended with the function:

def isDebug():
    withLettersOnly = lambda string: filter(lambda ch: ch.isalpha(), string)
    appName = os.path.basename(sys.argv[0])
    debugKey = "{appName}_DEBUG".format(appName = withLettersOnly(appName))

    return os.environ.get(debugKey, False)

and try/except statement will be replaced with the following code:

    try:
        main()
    except Exception, e:
        if isDebug():
            raise
        else:
            propagateToUser(e)
            sys.exit(1)

If the newly developed application is run from a file named bing-bang.py, environment variable bingbangpy_DEBUG set to a non-empty value will cause a raw Python's stack trace instead user-friendly error. The similar is true for a Python's script compiled using py2exe; guess a bound environment variable name.

Instead of Summary

1. The pattern code has been intentionally left primitive for one reason: to allow your to play around and find a suitable implementation;
2. Introduced own exception class could contain locals() and globals() of a corresponding layer; or system details. That totally depends on your fantasy;
3. The pattern works well for small scripts; and for "proof-of-concepts" applications which might be used in real-life until RTM. Avoid the approach in the case of more or less serious applications.

Python, argparse and Environment Variables

argparse more likely is the one of frequently used Python's libraries. It covers all standard cases out of the box. When a case goes beyond the box a developer is encouraged to extend the library with the specially provided API.

What to do when the argument is marked as required and its value is not changed during some quite long time? Make the argument's value persistent; store somewhere. Otherwise your application more likely has all chances to be recognized as unfriendly by an end-user. You may want to store a value of the argument somewhere in a configuration file. But what the file format should be? How to organize the file? Where should it be located? The are more questions than answers.

Why not to be able to pass the arguments to argparse's parser through the system environment? Such approach is widely recognized and makes your application easily scriptable.

Optional Arguments

When an argument is marked as optional a value from the system environment could be accessed by evaluating a default one:

parser.add_argument("-c", "--crt", type = str, default = os.environ.get("X509_CRT"), required = False, help = "Path to X509 Certificate")

Or in a bit complicated way when the value is not allowed to be empty:

parser.add_argument("-c", "--crt", type = str, default = os.environ.get("X509_CRT") or "~/.work/vpn.crt", required = False, help = "Path to X509 Certificate")

Required Arguments

Due to the design mandatory arguments in argparse library are not allowed to have default values.
There is an interface in argparse called Action which is associated with the argument being processed. The interface is intended to customize the way how an argument is processed/stored. Providing own implementation of the interface will allow you to look the desired value of the argument in system environment:

class FindValueInEnvironmentAction(argparse.Action):
    def __init__(self, varName, **kwargs):
        assert kwargs.get("required")
         
        valueFromEnv = os.environ.get(varName)
        requiredValue = True
        
        if valueFromEnv:
            kwargs["required"] = False
            kwargs["default"] = valueFromEnv
            
        argparse.Action.__init__(self, **kwargs)

    def __call__(self, parser, namespace, values, option_string):
        setattr(namespace, self.dest, values)
...
parser.add_argument("-c", "--crt", type = str, action = FindValueInEnvironmentAction, varName = "X509_CRT", required = True, help = "Path to X509 Certificate")

When the argument's value is found in the system environment variable scope, the built-in options in **kwargs are patched:

• required attribute is removed;
• default value is set to the read one.

These steps allow to pretend that an optional argument with a predefined default value is being processed. Here a value of the argument passed through the command line has a priority over a value set through "X509_CRT" environment variable.

I would also inject to our implementation a dictionary where to look up; it will allow us to cover the class with unit tests. And if you a user of Python3 feel free to try an alternative way.

Run Unit Test automatically for Python code

Have you ever thought how often you write the tests for the newly developed code? I believe that more likely -- every time. Even a tiny tool which will never leave the developer's sandbox, which is supposed to show that the implemented logic is correct might be called a "test"; it is not perfect, not maintainable, but it is intended to do at least one thing -- to check the code. I did so. Currently I prefer to write unit tests rather than such tiny tools; for the new as well as for the legacy code. If there is a file with the implementation -- there should be a corresponding file with the tests.

I am still Emacs user but sometimes for reasons unknown I use Sublime Text to create something using Python. So currently it is become boring a bit to:

1. Introduce the change in a file with a SUT;
2. Switch to the next tab in Sublime Text with a source code of the test;
3. Press a hot key to run the test to check the fixes.

I would prefer to replace these steps with the one. Here goes a simple straightforward solution to run unit tests for Python code in Sublime Text automatically; these tips are also applicable to other editors.

The test we have implemented for our code might look like:

import unittest
import hello as SUT

class TestHello(unittest.TestCase):
 def test_returnsHelloGretting(self):
  self.assertEqual("hello", SUT.greet())

The code we work on:

def greet():
 return "hello"

Way #0: Unit Tests inside

The most obvious way is to keep the SUT and the tests in the same file. In short I do not like this because of possible problems with the further maintainability.

Way #1: Native/Python

If we want to reduce the amount of switches between the tabs in the editor we need to make the SUT run the associated unit tests when the SUT is run as main python source. This could be done by introducing an intermediate helper layer to run the tests. Call it test.py:

import unittest
import inspect
import os

def main(**kwargs):
 ### stack[0] contains the frame information about the current function, since it was called first
 ### stack[1] contains the frame information about the caller
 callerFullPath = inspect.stack()[1][1]
 callerFileName = os.path.basename(callerFullPath)
 associatedTestFileName = "test_{0}".format(callerFileName)
 associatedTest = os.path.splitext(associatedTestFileName)[0]

 return unittest.main(associatedTest, verbosity = 2, **kwargs)

The function main() simply does the following:

1. Gets the file name of the SUT it was called from;
2. Composes a name of the file where the unit test for SUT is located in;
3. Redirects the call to the real test runner -- unittest.main()

... and slightly update contents of the file with the SUT:

def greet():
 return "hello"

if "__main__" == __name__:
 import test
 test.main()

Since now the each press of Ctrl-B (default hot key to run the build in Sublime Text) will lead to the run of unit tests.

Way #2: Custom build script

Sublime Text provides with a good ability to set up a custom build system for the projects a developer works on.

The final configuration might look like:

where python-wrapper.sh is a shell script to launch Python unit tests directly from Sublime Text on Ctrl-B. The script supposes that:

1. The tests are located in the same directory where the SUT is;
2. The file with a test has a prefix 'test_'.

... and it is compatible with bash:

#!/bin/bash

RUN_DIRECTORY="${1}"
SOURCE_NAME="${2}"
TEST_OF_SOURCE_NAME="test_${SOURCE_NAME}"

pushd "${RUN_DIRECTORY}"

if [ -e "${TEST_OF_SOURCE_NAME}.py" ]; then
 python -m unittest "${TEST_OF_SOURCE_NAME}"
else
 echo "[Warning] Associated test '${TEST_OF_SOURCE_NAME}' not found"
 python "${SOURCE_NAME}.py"
fi

RC=$?

popd

exit ${RC}

The simplicity of the solution might make you write and launch unit tests more often! But the solution should not be a reason not to run all unit tests periodically in the project.

Pass arguments to BaseHTTPRequestHandler

Each time when I face with the Python's built-in web-server (BaseHTTPServer) I feel a pain. The pain is caused by a strange architectural decision to pass a class as a request handler not an instance. At first glance it does not matter what kind of entity to pass. Since you can extend the default implementation with your logic. It still does not matter until a some moment. This moment happens when you want to have an externally configurable handler and/or you have to inject a bunch of settings. Currently there is no way to do it easily.

There is no easy way since the developer who created such design more likely was fell in love with Template Method pattern or was affected by some forbidden stuff :). Let's take a brief look how current Python's BaseHTTPRequestHandler's implementation works. Then let's try to answer the question how to pass arguments to BaseHTTPRequestHandler?

When a request comes to the server, the server creates an instance of BaseHTTPRequestHandler class. The newly created instance is initialized with a received request in raw format (say, as a plain not yet parsed text; when it comes finally to our handler, it is already split to the headers, body etc.). BaseHTTPRequestHandler's constructor dispatches an inner method (call it process_request()) responsible for an initial request handling; e.g. to determine a kind of the request (GET/POST/etc). After the request is recognized, a corresponding method do_[GET/POST/DELETE/HEAD/PUT]() is called from the self.

Seems very straightforward. But the following approach at least breaks the rule that one function should do one thing only. Constructor is responsible for object construction but not for serving business logic.

Let's see the following code. The handler is supposed to output current time and date in some format. With the current implementation the task could be implemented as:

def tellTheDate():
  import time
  return time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.wfile.write(tellTheDate())

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def main():
  host = ("localhost", 8080)
  server = BaseHTTPServer.HTTPServer(host, RequestHandler)
  server.handle_request()

The problem in the code above that it depends on the global scope. We need to avoid such dependency by injecting a required logic inside the handler. This is achieved by extending the handler's class.

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.logic()

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def tellTheDate(handler):
  import time
  currentDate = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
  handler.wfile.write(currentDate)

def main():
  host = ("localhost", 8080)
  handlerCls = RequestHandler

  handlerCls.logic = tellTheDate

  server = BaseHTTPServer.HTTPServer(host, handlerCls)
  server.handle_request()

After such modification the logic could be easily interchanged and could pretend as handler's built-in method. But since now it starts to break general encapsulation of extended BaseHTTPRequestHandler: the outer function has to know inner details of the class; e.g. to know how response is sent to the client. Also the implementation of handler is located across multiple locations: in the function with the logic and in the handler itself.

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def __init__(self, tellTheDate, *args):
    self.tellTheDate = tellTheDate
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args)
  
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.tellTheDate(self.wfile)

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def tellTheDate(output):
  import time
  currentDate = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
  output.write(currentDate)

def handleRequestsUsing(tellTheDateLogic):
  return lambda *args: RequestHandler(tellTheDateLogic, *args)

def main():
  host = ("localhost", 8080)

  handler = handleRequestsUsing(tellTheDate)
  server = BaseHTTPServer.HTTPServer(host, handler)
  server.handle_request()

The code above shows that a postponed (aka lazy) initialization mixed with Python's ability to setup a context where the function runs, makes the desired possible.

Make new ThinkPad's charge thresholds work in Debian/Ubuntu

Instead of Intro

ThinkPad`s have been always famous for the amazing hardware compatibility with Linux. But seems this time has gone and owners of new ThinkPad`s like ThinkPad X230, T430 and of some other may have started experiencing some uncomfortable changes. The one of the such changes is inability to use tp_smapi module to set battery thresholds.

Battery charge threshold might be important for you if you want extend the battery's health. They are used to keep batteries partially charged.

I am as an owner of brand new ThinkPad x230 run into the impossibility to load tp_smapi module for my primary Debian (Testing) GNU/Linux as well as to set above mentioned thresholds. Here will be a safe workaround to fix the regression. It must work as well for Ubuntu Linux.

For the previous versions of ThinkPad`s it was simply enough to put desired integers to /sys/devices/platform/smapi/BAT{0,1}/{start,stop}_charge_thresh files to make these thresholds work. Currently there are no such special files anymore; since tp_acpi loading is failed. At least for the newest ThinkPad`s. Fortunately there is an alternative way: to set these values directly through the kernel's ACPI subsystem not ThinkPad brand controller's interface.

There is a module called acpi_call which was originally designed for easy switching video adapters in dual graphics environments. It provides a convenient interface for sending any command to ACPI subsystem; literally "call ACPI for some request". But which command to send to request a change of charge thresholds? Actually it does not matter because you should not do it by yourself. There is a special tool called tpacpi-bat (the part of tpbattstat-applet) which will do it for you.

Solution: all steps together

The only things we need to make ThinkPad`s charge thresholds work in Debian/Ubuntu are:

1. Install acpi_call module;
2. Install tpacpi_bat script;
3. Configure thresholds setting on system boot-up;

acpi_call for Debian/Ubuntu

acpi_call is not included to standard Debian/Ubuntu repositories. The only way is to build it by yourself. Since I am a bit lazy about keeping in mind that with each kernel update I should rebuild acpi_call module I decided to write a simple script which will register and install it as a DKMS module.

The script could be retrieved from my git repository. The only requirement for running it is a root permission. If some packages required to build are missing you will be notified. Run it with 'run' argument and the script will do the following for you:

• Retrieve acpi_call sources from the main git repository;
• Register it as a DKMS module;
• Install acpi_module to your Debian/Ubuntu system.

tpacpi_bat

tpacpi_bat is also not included to standard Debian/Ubuntu repositories. It could be grabbed from the author's git repository. The script depends on Perl only and could be put anywhere on the system, e.g. to /usr/local/bin.

Thresholds on boot-up

To make charge thresholds set on system boot-up:

• acpi_call entry must be added to /etc/modules
• The following commands must be added to /etc/rc.local:

/usr/local/tpacpi-bat -v startChargeThreshold 0 N
/usr/local/tpacpi-bat -v stopChargeThreshold 0 K

where N and K are integers in percent of full battery capacity.

Enjoy! Thanks to this ThinkPad's batteries might be a bit healthy :).

Splitting OpenBox configuration to several files

Several weeks ago I decided to enhance performance of my Linux-powered ThinkPad x120e laptop. The one of the steps was switching from XFCE's window manager to a more lightweight and configurable one. I decided to give a try to OpenBox and since then I still like it.

Being a Software Developer I cannot stand pieces of software code which are large than ~15 lines. The one of the reasons is such code is hard to maintain. The same rule is applied for configuration files as well. OpenBox's configuration file represents itself a quite large XML file called rc.xml. And I really do not like this approach. I decided to find a way to split it out to several small pieces with the corresponding responsibility zones.

At first glance because of XML we could use a mechanism called External Entity. After this attempt was failed I discovered in the source code of OpenBox that processing of such entities is disabled and I get started to send a patch to enable it. But right away I found that XInclude mechanism is supported. After googling it was discovered that this feature was added in 2010. Now my OpenBox rc.xml is split to several files and looks like this:

<?xml version="1.0" encoding="UTF-8"?> <openbox_config xmlns="http://openbox.org/3.4/rc" xmlns:xi="http://www.w3.org/2001/XInclude">
  <xi:include href="resistance.xml"/>
  <xi:include href="focus.xml"/>
  <xi:include href="placement.xml"/>
  <xi:include href="theme.xml"/>
  <xi:include href="desktops.xml"/>
  <xi:include href="resize.xml"/>
  <xi:include href="margins.xml"/>
  <xi:include href="dock.xml"/>
  <xi:include href="keyboard.xml"/>
  <xi:include href="mouse.xml"/>
  <xi:include href="menu.xml"/>
  <xi:include href="applications.xml"/>
</openbox_config>

Note: because of syntax highlighter bug extra HTML tags may appear in this snippet. Please ignore them.

Where each include represents a corresponding configuration section. For example, resistance.xml:

<resistance>
    <strength>10</strength>
    <screen_edge_strength>20</screen_edge_strength>
</resistance>

As for me this is a very underrated feature which is not covered even in the official documentation. It makes the configuration components of OpenBox more granular and modular.

Note: Please be aware that OpenBox GUI configuration tools such as obconf may handle (and currently they do!) such files with XInclude`s.

How to replace screensaver in XFCE

Intro: XFCE's default screensaver

By default XFCE is shipped with a quite ascetic xscreensaver. Some people ma find it not attractive. So I do also. For example, if you install an alternative screen locking application called i3lock there are about no chances that it will be used by XFCE. Even there are no other alternatives installed. Because XFCE knows nothing about it. But user definitely knows and wants to replace the default screensaver with a preferred one.

The entry point for XFCE to screen lockers is a script called xflock{4}. All XFCE applications which need to start a screensaver or to lock the screen run it. (Note: Since version 4.10 it is shipped as a part of xfce4-session package.) E.g., XFCE's power manager will call xflock{4} when an action "Lock Screen" is specified for several ACPI events like "LID closed"; or when system goes suspend. The script itself launches only a default screensaver.

Possible alternatives

The ugly thing about this script is that it contains hardcoded list of screensavers. So if yours favorite screensaver is not listed there it won't be launched. The priority of these locking applications is:

1. xscreensaver
2. GNOME Screensaver
3. xlock
4. slock

And nothing else!

Solution

The straight way is to add an alternative screensaver is to hack xflock{4}'s contents. Replace:

for lock_cmd in \
  "xlock -mode blank" \
  "slock"
  do

with:

for lock_cmd in \
  "my-favorite-screensaver" \ # e.g. i3lock
  "xlock -mode blank" \
  "slock"
  do

But on the system's next update these changes more likely will be overrided with the default ones and yours screensaver will stop launching.

The best way of course is to extend xflock{4} and send the patch to upstream; but not this time :).

Another way is to pretend like we have a one of the above-mentioned screensavers which XFCE is familiar with:

1. Create a script which launches your screensaver. For example, with the following contents:

   #!/bin/sh
   if ! pidof i3lock > /dev/null 2>&1; then # launch once only
     /usr/bin/i3lock --dpms -c 000000
   fi
   

2. Make a symbolic link /usr/bin/slock to the newly created script;

After these steps there will be a new system's default screen locker.

Note: Do not forget to uninstall xscreensaver, gnome-screensaver, xlock from the system if there are any to prevent using of them because of their's higher priority.

Note: Please keep in mind about created symlink if once you decide to install an application with the same name.