Showing posts with label solution. Show all posts
Showing posts with label solution. Show all posts

Wednesday, September 4, 2013

Run Unit Test automatically for Python code

Have you ever thought how often you write the tests for the newly developed code? I believe that more likely -- every time. Even a tiny tool which will never leave the developer's sandbox, which is supposed to show that the implemented logic is correct might be called a "test"; it is not perfect, not maintainable, but it is intended to do at least one thing -- to check the code. I did so. Currently I prefer to write unit tests rather than such tiny tools; for the new as well as for the legacy code. If there is a file with the implementation -- there should be a corresponding file with the tests.

I am still Emacs user but sometimes for reasons unknown I use Sublime Text to create something using Python. So currently it is become boring a bit to:
  1. Introduce the change in a file with a SUT;
  2. Switch to the next tab in Sublime Text with a source code of the test;
  3. Press a hot key to run the test to check the fixes.
I would prefer to replace these steps with the one. Here goes a simple straightforward solution to run unit tests for Python code in Sublime Text automatically; these tips are also applicable to other editors.

The test we have implemented for our code might look like:

import unittest
import hello as SUT

class TestHello(unittest.TestCase):
 def test_returnsHelloGretting(self):
  self.assertEqual("hello", SUT.greet())

The code we work on:
def greet():
 return "hello"

Way #0: Unit Tests inside


The most obvious way is to keep the SUT and the tests in the same file. In short I do not like this because of possible problems with the further maintainability.

Way #1: Native/Python


If we want to reduce the amount of switches between the tabs in the editor we need to make the SUT run the associated unit tests when the SUT is run as main python source. This could be done by introducing an intermediate helper layer to run the tests. Call it test.py:

import unittest
import inspect
import os

def main(**kwargs):
 ### stack[0] contains the frame information about the current function, since it was called first
 ### stack[1] contains the frame information about the caller
 callerFullPath = inspect.stack()[1][1]
 callerFileName = os.path.basename(callerFullPath)
 associatedTestFileName = "test_{0}".format(callerFileName)
 associatedTest = os.path.splitext(associatedTestFileName)[0]

 return unittest.main(associatedTest, verbosity = 2, **kwargs)

The function main() simply does the following:
  1. Gets the file name of the SUT it was called from;
  2. Composes a name of the file where the unit test for SUT is located in;
  3. Redirects the call to the real test runner -- unittest.main()

... and slightly update contents of the file with the SUT:

def greet():
 return "hello"

if "__main__" == __name__:
 import test
 test.main()

Since now the each press of Ctrl-B (default hot key to run the build in Sublime Text) will lead to the run of unit tests.

Way #2: Custom build script


Sublime Text provides with a good ability to set up a custom build system for the projects a developer works on.

The final configuration might look like:

where python-wrapper.sh is a shell script to launch Python unit tests directly from Sublime Text on Ctrl-B. The script supposes that:

  1. The tests are located in the same directory where the SUT is;
  2. The file with a test has a prefix 'test_'.
... and it is compatible with bash:

#!/bin/bash

RUN_DIRECTORY="${1}"
SOURCE_NAME="${2}"
TEST_OF_SOURCE_NAME="test_${SOURCE_NAME}"

pushd "${RUN_DIRECTORY}"

if [ -e "${TEST_OF_SOURCE_NAME}.py" ]; then
 python -m unittest "${TEST_OF_SOURCE_NAME}"
else
 echo "[Warning] Associated test '${TEST_OF_SOURCE_NAME}' not found"
 python "${SOURCE_NAME}.py"
fi

RC=$?

popd

exit ${RC}

The simplicity of the solution might make you write and launch unit tests more often! But the solution should not be a reason not to run all unit tests periodically in the project.

Wednesday, May 29, 2013

Pass arguments to BaseHTTPRequestHandler

Each time when I face with the Python's built-in web-server (BaseHTTPServer) I feel a pain. The pain is caused by a strange architectural decision to pass a class as a request handler not an instance. At first glance it does not matter what kind of entity to pass. Since you can extend the default implementation with your logic. It still does not matter until a some moment. This moment happens when you want to have an externally configurable handler and/or you have to inject a bunch of settings. Currently there is no way to do it easily.

There is no easy way since the developer who created such design more likely was fell in love with Template Method pattern or was affected by some forbidden stuff :). Let's take a brief look how current Python's BaseHTTPRequestHandler's implementation works. Then let's try to answer the question how to pass arguments to BaseHTTPRequestHandler?

When a request comes to the server, the server creates an instance of BaseHTTPRequestHandler class. The newly created instance is initialized with a received request in raw format (say, as a plain not yet parsed text; when it comes finally to our handler, it is already split to the headers, body etc.). BaseHTTPRequestHandler's constructor dispatches an inner method (call it process_request()) responsible for an initial request handling; e.g. to determine a kind of the request (GET/POST/etc). After the request is recognized, a corresponding method do_[GET/POST/DELETE/HEAD/PUT]() is called from the self.

How BaseHTTPServer interacts with the handler


Seems very straightforward. But the following approach at least breaks the rule that one function should do one thing only. Constructor is responsible for object construction but not for serving business logic.

Let's see the following code. The handler is supposed to output current time and date in some format. With the current implementation the task could be implemented as:

def tellTheDate():
  import time
  return time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.wfile.write(tellTheDate())

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def main():
  host = ("localhost", 8080)
  server = BaseHTTPServer.HTTPServer(host, RequestHandler)
  server.handle_request()

The problem in the code above that it depends on the global scope. We need to avoid such dependency by injecting a required logic inside the handler. This is achieved by extending the handler's class.

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.logic()

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def tellTheDate(handler):
  import time
  currentDate = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
  handler.wfile.write(currentDate)

def main():
  host = ("localhost", 8080)
  handlerCls = RequestHandler

  handlerCls.logic = tellTheDate

  server = BaseHTTPServer.HTTPServer(host, handlerCls)
  server.handle_request()

After such modification the logic could be easily interchanged and could pretend as handler's built-in method. But since now it starts to break general encapsulation of extended BaseHTTPRequestHandler: the outer function has to know inner details of the class; e.g. to know how response is sent to the client. Also the implementation of handler is located across multiple locations: in the function with the logic and in the handler itself.

class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def __init__(self, tellTheDate, *args):
    self.tellTheDate = tellTheDate
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args)
  
  def do_GET(self):
    self.__process()

  def do_POST(self):
    self.__process()

  def __process(self):
    self.__setupLayout()
    self.tellTheDate(self.wfile)

  def __setupLayout(self):
    self.send_response(200)
    self.send_header("Content/Type", "text/plain")
    self.end_headers()

def tellTheDate(output):
  import time
  currentDate = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
  output.write(currentDate)

def handleRequestsUsing(tellTheDateLogic):
  return lambda *args: RequestHandler(tellTheDateLogic, *args)

def main():
  host = ("localhost", 8080)

  handler = handleRequestsUsing(tellTheDate)
  server = BaseHTTPServer.HTTPServer(host, handler)
  server.handle_request()

The code above shows that a postponed (aka lazy) initialization mixed with Python's ability to setup a context where the function runs, makes the desired possible.

Thursday, September 6, 2012

Make new ThinkPad's charge thresholds work in Debian/Ubuntu

Instead of Intro

ThinkPad`s have been always famous for the amazing hardware compatibility with Linux. But seems this time has gone and owners of new ThinkPad`s like ThinkPad X230, T430 and of some other may have started experiencing some uncomfortable changes. The one of the such changes is inability to use tp_smapi module to set battery thresholds.

Battery charge threshold might be important for you if you want extend the battery's health. They are used to keep batteries partially charged.

I am as an owner of brand new ThinkPad x230 run into the impossibility to load tp_smapi module for my primary Debian (Testing) GNU/Linux as well as to set above mentioned thresholds. Here will be a safe workaround to fix the regression. It must work as well for Ubuntu Linux.

For the previous versions of ThinkPad`s it was simply enough to put desired integers to /sys/devices/platform/smapi/BAT{0,1}/{start,stop}_charge_thresh files to make these thresholds work. Currently there are no such special files anymore; since tp_acpi loading is failed. At least for the newest ThinkPad`s. Fortunately there is an alternative way: to set these values directly through the kernel's ACPI subsystem not ThinkPad brand controller's interface.

There is a module called acpi_call which was originally designed for easy switching video adapters in dual graphics environments. It provides a convenient interface for sending any command to ACPI subsystem; literally "call ACPI for some request". But which command to send to request a change of charge thresholds? Actually it does not matter because you should not do it by yourself. There is a special tool called tpacpi-bat (the part of tpbattstat-applet) which will do it for you.

Solution: all steps together

The only things we need to make ThinkPad`s charge thresholds work in Debian/Ubuntu are:
  1. Install acpi_call module;
  2. Install tpacpi_bat script;
  3. Configure thresholds setting on system boot-up;

acpi_call for Debian/Ubuntu

acpi_call is not included to standard Debian/Ubuntu repositories. The only way is to build it by yourself. Since I am a bit lazy about keeping in mind that with each kernel update I should rebuild acpi_call module I decided to write a simple script which will register and install it as a DKMS module.

The script could be retrieved from my git repository. The only requirement for running it is a root permission. If some packages required to build are missing you will be notified. Run it with 'run' argument and the script will do the following for you:
  • Retrieve acpi_call sources from the main git repository;
  • Register it as a DKMS module;
  • Install acpi_module to your Debian/Ubuntu system.

tpacpi_bat

tpacpi_bat is also not included to standard Debian/Ubuntu repositories. It could be grabbed from the author's git repository. The script depends on Perl only and could be put anywhere on the system, e.g. to /usr/local/bin.

Thresholds on boot-up

To make charge thresholds set on system boot-up:
  • acpi_call entry must be added to /etc/modules
  • The following commands must be added to /etc/rc.local:
/usr/local/tpacpi-bat -v startChargeThreshold 0 N
/usr/local/tpacpi-bat -v stopChargeThreshold 0 K

where N and K are integers in percent of full battery capacity.


Enjoy! Thanks to this ThinkPad's batteries might be a bit healthy :).

Sunday, July 22, 2012

Splitting OpenBox configuration to several files

Several weeks ago I decided to enhance performance of my Linux-powered ThinkPad x120e laptop. The one of the steps was switching from XFCE's window manager to a more lightweight and configurable one. I decided to give a try to OpenBox and since then I still like it.

Being a Software Developer I cannot stand pieces of software code which are large than ~15 lines. The one of the reasons is such code is hard to maintain. The same rule is applied for configuration files as well. OpenBox's configuration file represents itself a quite large XML file called rc.xml. And I really do not like this approach. I decided to find a way to split it out to several small pieces with the corresponding responsibility zones.

At first glance because of XML we could use a mechanism called External Entity. After this attempt was failed I discovered in the source code of OpenBox that processing of such entities is disabled and I get started to send a patch to enable it. But right away I found that XInclude mechanism is supported. After googling it was discovered that this feature was added in 2010. Now my OpenBox rc.xml is split to several files and looks like this:

<?xml version="1.0" encoding="UTF-8"?> <openbox_config xmlns="http://openbox.org/3.4/rc" xmlns:xi="http://www.w3.org/2001/XInclude">
  <xi:include href="resistance.xml"/>
  <xi:include href="focus.xml"/>
  <xi:include href="placement.xml"/>
  <xi:include href="theme.xml"/>
  <xi:include href="desktops.xml"/>
  <xi:include href="resize.xml"/>
  <xi:include href="margins.xml"/>
  <xi:include href="dock.xml"/>
  <xi:include href="keyboard.xml"/>
  <xi:include href="mouse.xml"/>
  <xi:include href="menu.xml"/>
  <xi:include href="applications.xml"/>
</openbox_config>

Note: because of syntax highlighter bug extra HTML tags may appear in this snippet. Please ignore them.

Where each include represents a corresponding configuration section. For example, resistance.xml:

<resistance>
    <strength>10</strength>
    <screen_edge_strength>20</screen_edge_strength>
</resistance>

As for me this is a very underrated feature which is not covered even in the official documentation. It makes the configuration components of OpenBox more granular and modular.

Note: Please be aware that OpenBox GUI configuration tools such as obconf may handle (and currently they do!) such files with XInclude`s.

Wednesday, May 30, 2012

How to replace screensaver in XFCE

Intro: XFCE's default screensaver

By default XFCE is shipped with a quite ascetic xscreensaver. Some people ma find it not attractive. So I do also. For example, if you install an alternative screen locking application called i3lock there are about no chances that it will be used by XFCE. Even there are no other alternatives installed. Because XFCE knows nothing about it. But user definitely knows and wants to replace the default screensaver with a preferred one.

The entry point for XFCE to screen lockers is a script called xflock{4}. All XFCE applications which need to start a screensaver or to lock the screen run it. (Note: Since version 4.10 it is shipped as a part of xfce4-session package.) E.g., XFCE's power manager will call xflock{4} when an action "Lock Screen" is specified for several ACPI events like "LID closed"; or when system goes suspend. The script itself launches only a default screensaver.

Possible alternatives

The ugly thing about this script is that it contains hardcoded list of screensavers. So if yours favorite screensaver is not listed there it won't be launched. The priority of these locking applications is:
  1. xscreensaver
  2. GNOME Screensaver
  3. xlock
  4. slock
And nothing else!

Solution

The straight way is to add an alternative screensaver is to hack xflock{4}'s contents. Replace:

for lock_cmd in \
  "xlock -mode blank" \
  "slock"
  do

with:

for lock_cmd in \
  "my-favorite-screensaver" \ # e.g. i3lock
  "xlock -mode blank" \
  "slock"
  do

But on the system's next update these changes more likely will be overrided with the default ones and yours screensaver will stop launching.
The best way of course is to extend xflock{4} and send the patch to upstream; but not this time :).
Another way is to pretend like we have a one of the above-mentioned screensavers which XFCE is familiar with:
  1. Create a script which launches your screensaver. For example, with the following contents:
    #!/bin/sh
    if ! pidof i3lock > /dev/null 2>&1; then # launch once only
      /usr/bin/i3lock --dpms -c 000000
    fi
    
  2. Make a symbolic link /usr/bin/slock to the newly created script;
After these steps there will be a new system's default screen locker.

Note: Do not forget to uninstall xscreensaver, gnome-screensaver, xlock from the system if there are any to prevent using of them because of their's higher priority.

Note: Please keep in mind about created symlink if once you decide to install an application with the same name.

Tuesday, May 1, 2012

Some TrueCrypt sugar for Linux

Intro

Currently we have a fast growing trend called "Clouds". In the Clouds we can store about everything and be sure that we are able to get it back anywhere and anytime. The only thing is Clouds do not belong to us. That's why we should think critical when we upload a very-very private data there. This way of thinking leads us to use various file encryption tools like encfs, TrueCrypt and other.

I like TrueCrypt. It is installed on my every Linux workstation. But sometimes I am not the only user of such workstations. There could be several concurrent users who are able to browse computer's filesystem. Do you remember the default behavior of TrueCrypt when it is asked (I mean double-click inside File Manager) to mount an encrypted volume? Right! By default it mounts a specified container somewhere to /mnt (/media) directory which could be browsed by other online users. That means they are able to see my private data stored in the mounted TrueCrypt container! Nein, I do not want this! I want to make double-click on a TrueCrypt container and be the only viewer of it's contents.

That's why I have written a kind of mounter of TrueCrypt containers which connects decrypted volume to user's home folder.

In Action

Suppose that our encrypted volumes have ".private" extension in the filename. To make them always be opened with the mounter we should make a corresponding file association:
Make TrueCrypt container association with the truecrypt-mount application
TrueCrypt mounter (simple wrapper over TrueCrypt) is located in /usr/bin:
Specify an absolute path to truecrypt-mount application
Then TrueCrypt asks for the password:
Enter container's password to unlock the data
User-Locally (!) mounted container is immediately shown in a new window of File Manager. Please notice the path where the container is mounted to.
Immediate browsing of mounted TrueCrypt volume
If there is an attempt to mount already connected TrueCrypt's volume with truecrypt-mount application there will be a corresponding error message:
Already mounted TrueCrypt volume error message


In details

truecrypt-mount as I have already mentioned above is a simple wrapper over TrueCrypt itself. The wrapper is just a POSIX shell script which redirects volume mount requests from the user to TrueCrypt application with the "correct" arguments.

truecrypt-mount is configured through /etc/default/truecrypt file. The following options are supported:

TRUECRYPT_BINARY -- path where TrueCrypt itself is installed. By default points to /usr/bin/truecrypt.

USER_MOUNT_ROOT -- path where will be created user-specific mounts.

EXPLORE_MOUNTED_VOLUME -- boolean value which specifies should be a newly mounted TrueCrypt volume shown in the File Browser or not.

ERROR_REPORTING_CHANNEL -- specifies how errors while mounting are shown.

Download

The wrapper is available for download from the git repository http://git.thekondor.net
truecrypt-mount is a part of truecrypt-extra package since there is not the only thing how the experience with TrueCrypt could be improved (see below).

The application as well as the package is distributed in the terms on GNU GPL v3.0+ license.

As a benefit

Exploring volume in XFCE

Have you ever tried to launch TrueCrypt from a command line with '--explore' option being in XFCE? If you have, you must have also probably received an error that "Nautilus" is not found. Accordingly to the source codes of TrueCrypt it knows nothing about Thunar! Then you possibly tried to work around the problem by creating a fake Nautilus "application".

Thanks to EXPLORE_MOUNTED_VOLUME option for truecrypt-mount there is no need in such workarounds anymore; we get it just for free! By calling /usr/bin/xdg-open there will be opened default File Manager.

Sleep means Sleep

That's all about being kinda paranoid :). What if a laptop while hibernated is stolen? What if there are several mounted TrueCrypt volumes? That means that a Bad Guy having a hibernation image is able to retrieve sensitive data from those volumes because they were not disconnected! To prevent this kind of hole user should always unmount its TrueCrypt volumes when laptop is sent to sleep/hibernate mode.

truecrypt-extra package provides PM-Utils script 20_unmount-truecrypt-containers which unmounts ALL connected TrueCrypt volumes in the user's system when it goes sleep/hibernate. This behavior could be disabled by UNMOUNT_ON_SLEEP option.

Not a summary

I still have several ideas about improving TrueCrypt experience in Linux which might be implemented in the terms on truecrypt-extra package. So stay tuned and send your feedback regarding the package if any.

Saturday, December 3, 2011

Reduce Linux laptop's backlight on boot up

I like Linux. Especially for being able to create graceful workarounds for the things we get accustomed in a real life.

Being a Linux-powered laptop owner I have always been annoyed by sharp backlight change when Linux boots up. I do not like when a backlight is set to the maximum one and there are no ways to affect these settings on. So I did a very funny trick.

System's backlight (aka "display's brightness") in Linux could be changed through the special /sys/class/backlight/acpi_video0/brightness file. When an integer is written Linux immediately changes laptop's backlight level. E.g. the following command:
# echo 10 > /sys/class/backlight/acpi_video0/brightness

will set display's brightness to 10 points.

The first steps of Linux boot up are made with initramfs scripts which are executed one by one. When system-required-specific modules are loaded laptop's backlight level dramatically raises up. So we can think about this point in time as a moment when the user is able to tell Linux not to do so; or at least set it explicitly to a preferred value. That moment in time in the terms of initrams-tools could be called "premount".
#!/bin/sh

### file: /etc/initramfs-tools/scripts/local-premount/backlight_level
### this file must have an executable bit.
###
### Author: Andrew Sichevoi
### Please feel free to send your bug reports to http://blog.thekondor.net

PREREQ=""
prereqs()
{
  echo "${PREREQ}"
}

case "${1}" in
  prereqs)
           exit 0;
           ;;
esac

. /scripts/functions

DEFAULT_BACKLIGHT_LEVEL=10
BACKLIGHT_LEVEL=
for arg in $(cat /proc/cmdline); do
  case ${arg} in
    backlight_level=*)
                       BACKLIGHT_LEVEL=${arg#backlight_level=}
                       ;;
  esac
done

if [ -z ${BACKLIGHT_LEVEL} ]; then
   log_warning_msg "Using default backlight level: '${DEFAULT_BACKLIGHT_LEVEL}'"
   BACKLIGHT_LEVEL=${DEFAULT_BACKLIGHT_LEVEL}
fi

echo ${BACKLIGHT_LEVEL} > /sys/class/backlight/acpi_video0/brightness

So here we go:
  1. Add the below-mentioned backlight_level script to /etc/initramfs-tools/scripts/local-premount directory, make it executable;
  2. Update initramfs using update-initramfs command (Linux distribution specific; at least that works for Debian and Ubuntu) to include the script to the initrd image;
  3. Set passing backlight_level=X (where X is a preferred backlight level) boot option in your's grub.cfg.
and reboot to see the changes in action.

The script just looks up backlight_level kernel boot option and writes (if any, or default one otherwise) this value to the brightness file for immediate laptop's brightness update.

Simple? I think yes. Does it work? Definitely. Perfect? Not at all: I believe that system's boot's up settings should not differ from main DE's ones: they should be configured through the single entry point not several ones. Anyway this scripts makes feel us that we able to control Linux as well as we have a new task to think about :). So stay tuned!

Friday, November 11, 2011

Python, make ConfigParser aware of spaces

There is a wonderful Python's module called ConfigParser which allows to process .ini-style configuration files easily. I prefer to use it everywhere rather than spend the time to implement my own solution. Recently there was a bug received that values with leading and trailing spaces are read incorrectly: spaces are lost. This might be important for cases when an application is sensitive for such values; e.g.: for passwords.

It was discovered that current Python's ConfigParser implementation cannot be tuned up not to strip values while reading a configuration. Also there was a corresponding issue found with an attached patch. Unfortunately the patch has not been applied to public available Python builds yet. Definitely it is absolute not convenient to patch Python everywhere where yours application is run.

The solution is not to lose leading and trailing spaces by wrapping them for quotes. Here is a helping code snippet to solve this issue:

class SpaceAwareConfigParser(ConfigParser.ConfigParser):
    def __init__(self, **args):
        KEEP_SPACES_KEYWORD = "keep_spaces"
        
        self.__keep_spaces = args.get(KEEP_SPACES_KEYWORD, True)
        args.pop(KEEP_SPACES_KEYWORD)

        ConfigParser.ConfigParser.__init__(self, **args)

    def get(self, section, option):
        value = ConfigParser.ConfigParser.get(self, section, option)
        if self.__keep_spaces:
            value = self._unwrap_quotes(value)

        return value

    def set(self, section, option, value):
        if self.__keep_spaces:
            value = self._wrap_to_quotes(value)

        ConfigParser.ConfigParser.set(self, section, option, value)        

    @staticmethod
    def _unwrap_quotes(src):
        QUOTE_SYMBOLS = ('"', "'")
        for quote in QUOTE_SYMBOLS:
            if src.startswith(quote) and src.endswith(quote):
                return src.strip(quote)

        return src

    @staticmethod
    def _wrap_to_quotes(src):
        if src and src[0].isspace():
            return '"%s"' % src

        return src

Overridden get() method removes quotes if any. So for ConfigParser's client that is transparent if the option has a value with quoted spaces or not; double and single quotes are supported. set() does vice versa.

So it is enough to replace instantination of ConfigParser in your Python's code just with SpaceAwareConfigParser one and it should work like expected.

Sunday, May 22, 2011

Python, imp.load_source() trap

While I have been writing a hook for WebApy lightweight RESTful Python webserver -- the recent a project of mine, I got ran into the funny (actually it wasn't; since it was hard enough to debug) issue related to loading of Python-app addons. As far as you know (or not; if you already have taken a look at sources), there is Python's standard library's 'imp' module is used to load hook files. "imp.load_source()" if to be more precised.

Here is an example of the problem you may get into in Python while loading modules with load_source() function of imp module.

Lets see a simplified example of what where I ran into. Suppose our Python application supports external addons (aka plugins/extensions). Each addon must provide a class named "Addon" with implemented "runLogic()" method. Addons have priorities -- from 1 to N (N > 1); priority set to "1" is a highest one. Suppose several of addons also implement internal Helper class for low-level dirty work.

Addon #1 (file: addon-1.addon.py)
class Helper(object):
    """ Internal, addon-specific helper class """
    def __init__(self, msg):
        self.__msg = msg

    def help(self):
        print self.__msg
        
class Addon(object):
    """ Addon entry point class """
    PRIORITY = 1
    
    def runLogic(self):
        addonHelper = Helper("module 1")
        addonHelper.help()

and Addon #2 (file: addon-2.addon.py)
class Helper(object):
    """ Internal, addon-specific helper class """
    def help(self):
        print "module 2"

class Addon(object):
    """ Addon entry point class """
    PRIORITY = 2
    
    def runLogic(self):
        addonHelper = Helper()
        addonHelper.help()

Addons are handled inside the main application through the class named "AddonLoader". "AddonLoader" is initialized with the only argument -- filename pattern of modules; since our modules are called addon-1.addon.py and addon-2.addon.py respectively the aforementioned pattern could be '*.addon.py'. Also "AddonLoader" provides with the only public method "runMoreImportantAddon()" which executes an addon with the highest priority. Addons handling is implemented through Python's 'imp' module:

import glob  ### To find addons on local filesystem
import imp   ### To load addons

class AddonLoader(object):
    def __init__(self, addonPattern):
        self.__addonPattern = addonPattern

    def runMoreImportantAddon(self):
        """ Runs an addon with the highest priority (determined by PRIORITY property)  """
        moreImportantAddon = self.__getMoreImportantAddonInstance()
        moreImportantAddon.runLogic()

    def __getMoreImportantAddonInstance(self):
        """ Returns an addon instance with the highest (1 -- high, 10 -- low) priority """
        
        availableAddons = self.__enumerateAddonsOnFileSystem()
        sortedAvailableAddons = sorted(availableAddons,
                                       key = lambda addon: addon.PRIORITY)

        return sortedAvailableAddons[0]()

    def __enumerateAddonsOnFileSystem(self):
        """ Load addons by specified pattern and returns a list with them """
        
        addons = list()
        
        for addonPath in glob.glob(self.__addonPattern):
            addon = imp.load_source("addon", addonPath)
            addons.append(addon.Addon)

        return addons

def main():
    addonLoader = AddonLoader("*.addon.py")
    addonLoader.runMoreImportantAddon()

if "__main__" == __name__:
    main()

So could you predict what the output will be when main app is ran? I bet that probably not. There will be an exception that Helper class could not be instantiated because of invalid passed parameters amount. Was it expected?

No, it was not; at least for me. The problem of this code is hidden in "name" argument of imp.load_source()'s function. For each enumerated addon it is still the same (set explicitly to "addon"); on the each iteration all already loaded classes are overwritten. On first iteration we extract and keep a reference to Addon #1 (do not forget that it uses Helper #1 class). On the second (final) iteration a reference to Addon #2 (it depends on Helper #2 class) is taken. Since loaded addons are set to have the same name ("addon"), on the final iteration we see that Helper #1 is overwritten with Helper #2 in the namespace of "addon". And when Addon #1 is being instantiated it is calls for Helper class, but as you remember it was overwritten and does not take any parameters in the constructor. Here we get an exception.

The solution of the problem is not to pass constant name to load_source() function. Just replace "addon" there with the call of the function which returns an unique name of module to load. Names must be unique!

Since the official documentation does say nothing about uniqueness of "name" argument, I wonder why there is no hint about the trap you could get into while using it?

Sunday, May 1, 2011

Broken Java networking in Debian

Playing around with Java I ran into the problem well known as Bug #560044. Seems that the problem appears for Debian GNU/Linux only. The essence of the issue is: when you run any Java application that requires network access (by HTTP or by other protocol; it does not matter) you will (as well as the running application itself) receive the following error. Or a very similar one:

Exception in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

This stack trace says nothing specific about occurred error except that a connection with a remote server cannot be established. The problem is at the bottom of explicitly set in Debian bindv6only kernel parameter. When it turned on (set to "1" in terms of sysctl) it means that network-requesting application will receive IPv6 socket by default not IPv4. To be able to use IPv4 socket an application should open it explicitly. Since some applications rely on the host system's settings it leads that they may fail while network access (details).

The solution is pretty easy: to disable explicit IPv6 binding in your Debian GNU/Linux. This can be done by the command execution:

echo 0 | sudo tee -a /proc/sys/net/ipv6/bindv6only 
This snippet will fix the problem for the current session only. To make the changes in Debian permanent the another one should be performed (proposed by Heikki Henriksen):
sudo sed -i 's/net.ipv6.bindv6only\ =\ 1/net.ipv6.bindv6only\ =\ 0/' /etc/sysctl.d/bindv6only.conf && sudo invoke-rc.d procps restart

After this fix (I believe that it is still a w/a) Java applications can have network access and not raise incomprehensible exceptions anymore.

Wednesday, June 2, 2010

APT suggests to remove (autoremove) required packages

After another packages update (or after my attempts to remove Evolution out from the system, I do not know exactly) on my Debian Squeeze, APT began to suggest me to remove 'non-required' packages using 'autoremove' option like this:

The following packages were automatically installed and are no longer required: totem-common libempathy-gtk28 libaprutil1-ldap vinagre odbcinst seahorse-plugins libgeoclue0 xdg-user-dirs gnome-user-share libgalago3 libtelepathy-farsight0 gconf-defaults-service guile-1.8-libs unixodbc
...
Use 'apt-get autoremove' to remove them.

It'd be Ok if packages to remove were really non-required for my system. But they ain't! W/o them my Gnome desktop would be definitely broken. Seems that something went wrong with 'Depends' field of some package.

This had been annoying me until I found a solution:

# apt-get install <list-of-packages-suggested-to-remove>

All above-mentioned packages were installed automatically due 'Depends' field of some (name it 'primary') very important package. Hence APT believes that it's Ok to remove them out when the primary package is purged. By explicitly using 'apt-get install' we tell APT that these packages ain't automatically installed anymore but manually. Pretty easy, yeah?

Alternative ways

  • Use 'aptitude unmarkauto' to mark packages manually installed;
  • Update /etc/apt/apt.conf with debug::pkgAutoRemove=false key to disable APT's autoremoving feature.
Enjoy.

Saturday, May 22, 2010

Twitter and my solution for Debian + Thinkpad issues

Having the latest Debian Squeeze (kernel 2.6.32) I got in some issues (hm... Seems that for some time I'm going to write about Debian and my Thinkpad only :)) with my laptop.

The common issues that I (and Google proves that not only my Thinkpad T410 run into them) had were:
  1. LCD brightness up/down doesn't work with loaded up ``thinkpad_acpi'' module; ``hotkeys'' module param does not help. Even ``acpi_listen'' does not show about corresponding events.
  2. On resume after suspend/hibernate USB goes to sleep or even become disabled;
  3. On resume after suspend/hibernate the system receives a kernel trace error with registers dump.
And I was playing around with them. But nothing really helps: seemed that the issues are more deeper than they look at first glance. These unanswered questions were blocking me from comfortable using of my device which I really like.

Meanwhile there was an announce about Linux 2.6.34 kernel with the fix of famous bug #12309...

So I decided to try it out but unfortunately there were (still are) no Debian packages with the latest kernel sources for Squeeze/Sid. Only vanilla kernel w/o corresponding Debian patches. But Debian Experimental already has it! So what we do:

  1. Download linux-kbuild-2.6.34 from http://ftp.de.debian.org/debian/pool/main/l/linux-kbuild-2.6/ and install it;
  2. Download linux-source-2.6.34 from http://ftp.de.debian.org/debian/pool/main/l/linux-2.6/ and install it;
  3. Go to /usr/src/ and extract .tar.gz with linux sources; then go to the newly created dir with the linux 2.6.34 sources;
  4. Copy current copy config (has a name like ``config-2.6.32-686'') from /boot as .config to the current directory;
  5. Build the kernel with 'make-kpkg' command as described in http://www.debian-administration.org/article/Building_your_own_v2.6.x_Kernel_in_the_Debian_manner.
You will be prompted to answer several questions about using new features (and devices support, of course) added to the new version of kernel. When this kind of survey is finished you can go to take some cups of coffee.


After booting the new Debian kernel I got that those Thinkpad issues became resolved for me.

P.S. BTW, I finally decided to start my personal twitter. Sometimes I have a lot ideas and thoughts I want to share but they ain't so big or not so significant or even I do not have enough time to make standalone posts about them. Hence such 'dumps' are go to my twitter; I hope they are useful and interesting.


Thursday, May 20, 2010

Debian Lenny + Thinkpad T410: network adapter issue

Since I chose Debian to be my primary and favorite Linux distribution on my new Lenovo Thinkpad T410, I agreed to myself to spend my time to solve various specific issues related to it.

So the first issue I met was that the Debian Lenny installation system didn't manage to find my Thinkpad's Intel 1000 ethernet adapter. Google 'said' that appropriate driver for the adapter was 'e1000e'. Manual 'e1000e' module loading (through 'modprobe' on another virtual console) didn't help: seems the driver is slight outdated and doesn't support newer Intel's network adapter.

The solution is simple: to use an alternative disk image made by Kenshi Muto to install Debian Lenny. This ISO can be downloaded from http://kmuto.jp/debian/d-i/.
Thankfully to the latest version of kernel, Debian installation system will detect Thinkpad's network adapter and the user will be able to continue installation from Internet not from CD/DVD only.

Or there is an alternative way: to compile the latest version of Intel's driver and load it from USB-stick during the installation.