CyanogenMod 11 on a Samsung Galaxy Note (N7000) – Nope!

It’s important to also publish negative results. So this as a memo to myself and a recommendation for others who would like to install CyanogenMod 11 on a Samsung Galaxy Note (N7000). Don’t do it. I installed it using the nightly build cm-11-20150817-NIGHTLY-n7000.zip and the Google Apss gapps-kk-20140606-signed.zip. The device reacted very slowly afterward and raised often error messages. I went back and installed CM 10 using cm-10.1.3-n7000.zip and gapps-jb-20130813-signed.zip. Back to happy.

Merging mp3 files

Sometimes things are easier than ones might have thought. I wanted to merge some mp3 files in the shell and found that this can simply be done using cat.

$ cat my_file_1.mp3 my_file_2.mp3 my_file_3.mp3 > combined_files.mp3

Forgiving players like mplayer can handle the resulting files without any issue. Still, it’s recommended to fix the file header e.g. using avconv.

$ avconv -i combined_files.mp3 -acodec copy combined_files_fixed.mp3

Setting up ownCloud

I recently set up ownCloud (version 6.0.2) at my web hoster domainfactory. While the actual installation is quite straight forward I bumped into some minor problems.

At first I set up subdomain the ownlcoud.notmydomain.com and set the path to /my/dummy/path/owncloud in the domainfactory dash board. I SSHed to the server of domainfactory, downloaded and installed ownCloud’s files:

$ cd /my/dummy/path/
$ wget http://download.owncloud.org/community/owncloud-6.0.2.tar.bz2
$ tar xfj owncloud-6.0.2.tar.bz2

This generated the folder owncloud which contains all the required files in the configured location.

When I opend ownlcoud.notmydomain.com in a webbrowser I got the following error:

Warning: Unexpected character in input: '\' (ASCII=92) state=1 in /my/dummy/path/owncloud/index.php on line 33

Warning: Unexpected character in input: '\' (ASCII=92) state=1 in /my/dummy/path/owncloud/index.php on line 33

Parse error: syntax error, unexpected T_STRING in /my/dummy/path/owncloud/index.php on line 33

I figured out that this can be fixed by using a newer PHP version so I switched from 5.2.17 to 5.4.9 using the dash board of my domainfactory account. When I visited the subdomain in the webbrowser the following welcome screen was presented:

owncloud1

By clicking “Advanced” the database options are shown and can be configured.

owncloud2

Per default sqlite is used but I wanted to use MySQL.

owncloud3

I entered the details.

owncloud4

And clicked “Finish setup”. This the dash board of my ownCloud instance showed that there are some folders and files had been parked for me already.

owncloud5

I clicked the plus at the bottom left and activated the Encryption app for server-side file encryption. To initiate the key I had to logout and login again.

owncloud6

Additionally if I went to the settings and activated forcing of SSL-encrypted login.

owncloud7

When uploading the first files I saw that the upload limit was 8Mb. To change this the values upload_max_filesize and post_max_size in the php.ini had to be adapted. This can be done via the domainfactory dash board. And voilà, here is the dropbox/google drive/$YourNSAaccessibleCloudStorage replacement!

Python packaging error

Let’s face it – the Python packaging is not necessary famous to be one of the best *cough*. But the following issue was purely my fault: I made a stupid error when generating a Python package. I had a constellation like this:

my_dummy_project/
├── bin
│   └── collide.py
├── collide
│   ├── a_module.py
│   ├── another_module.py
│   └── __init__.py
├── README.txt
└── setup.py

I generated a package based on this calling python setup.py sdist. The collide.py script had an import statement to load a module stored in the collide folder. But as Python looks in the folder of the script for a module with such name first before checking other parts of the Python path it was reading the very same script again as module and raised an ImportError. It took me some time to figure out that this error occurs as the names of the script and the library folder are the same. I circumvent this by renaming the folder collide to collidelib and adapting the import statement accordingly. Memo to self – do not name the script in the bin folder of a package the same as the folder with the modules!

pysam for Python 3.2, on Ubuntu – mainly a Cython problem

Update: With the recently released pysam version 0.7.4 this works now also for Python 3.3.

I wanted to install pysam for Python 3.2 (actually, I wanted to install it for Python 3.3 but this is not supported so far) on a machine with Ubuntu 12.10 and ran into some trouble. The solution might be helpful for other pysam users and similar issues. What happened? I downloaded pysam 0.7 and tried to build it:

$ wget https://pysam.googlecode.com/files/pysam-0.7.tar.gz
$ tar xfz pysam-0.7.tar.gz
$ cd pysam-0.7/
$ python3.2 setup.py build
Traceback (most recent call last):
  File "setup.py", line 123, in 
    from Cython.Distutils import build_ext
ImportError: No module named Cython.Distutils

As you can see I got a error message telling me that the Cython.Distutils module is not available. I had the Ubuntu package cython installed and when performing the installation with python2.7 this problem did not occure. The solution is to install Cython manually for Python 3.2. For this it is necessary to have the packages python3.2-dev installed. Otherwise you will get the following error:

fatal error: Python.h: No such file or directory

So make sure you have this package (and of course python3.2) is installed

$ sudo apt-get update
$ sudo apt-get install python3.2 python3.2-dev

Now you download and install Cython:

$ wget http://cython.org/release/Cython-0.17.4.tar.gz
$ tar xfz Cython-0.17.4.tar.gz
$ cd Cython-0.17.4
$ python3.2 setup.py install --user

I use the –user flag to install Cython module (and later pysam) only locally to my user directory. If you prefer to install it system wide add sudo in front of the call and leave the flag away. Once Cython is available for Python 3.2 the installation of pysam works without any problems

$ cd pysam-0.7/
$ python3.2 setup.py build
[...]
$ python3.2 setup.py install --user
[...]

Now you should be able to import the pysam module in Python 3.2:

$ python3.2
Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
>>> dir(pysam)
['AlignedRead', 'Fastafile', 'IndexedReads', 'IteratorColumn', 'IteratorRow', 'Pileup', 'PileupColumn', 'PileupProxy', 'PileupRead', 'SAMTOOLS_DISPATCH', 'Samfile', 'SamtoolsDispatcher', 'SamtoolsError', 'Tabixfile', 'VCF', 'VCFRecord', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__samtools_version__', '__version__', 'asBed', 'asGTF', 'asTuple', 'asVCF', 'bam2fq', 'calmd', 'cat', 'cmd', 'csamtools', 'ctabix', 'cvcf', 'depth', 'faidx', 'fixmate', 'flagstat', 'idxstats', 'index', 'key', 'merge', 'mpileup', 'options', 'os', 'parser', 'phase', 'reheader', 'rmdup', 'samimport', 'sort', 'sys', 'tabix_compress', 'tabix_index', 'tabix_inplace_iterator', 'tabix_iterator', 'targetcut', 'tview', 'version', 'view']

Using the pipeline module ruffus with Python 3.2

The Python module ruffus is a great companion for setting up pipelines easily and quickly. Unfortunately the development has been focused on Python 2.x … until recently. There are some recent adaptations to get it running under Python 3.x but there are some modifications needed.

At first download the development version of ruffus:

$ svn checkout http://ruffus.googlecode.com/svn/trunk/ ruffus-read-only

Then use the 2to3 tool to for the code conversion:

$ 2to3 -w ruffus-read-only/

Let’s check if this works by running a small script:

$ cd ruffus-read-only/
$ emacs -nw test.py
$ cat test.py
from ruffus import *

def first_task():
    print("One")

@follows(first_task)
def second_task():
    print("Two")

pipeline_run(second_task)
$ python --version
Python 3.2
$ python test.py
Traceback (most recent call last):
  File "test.py", line 10, in 
    pipeline_run(second_task)
  File "[...]/ruffus-read-only/ruffus/task.py", line 2675, in
pipeline_run
    pool_func = imap
NameError: global name 'imap' is not defined

As you can see 2to3 did not do the full job. But the code needs only a small adaptation to be Python 3.x compatible (it is a tiny problem due to changes in itertools):

$ cp ruffus/task.py ruffus/task.py.org
$ emacs -nw ruffus/task.py
$ diff ruffus/task.py.org ruffus/task.py
2675c2675
<         pool_func = imap
---
>         pool_func = map

$ python test.py
One
    Job completed
Completed Task = first_task
Two
    Job completed
Completed Task = second_task 

Now it works!

Testing Django – part 4 – setting up a dedicated CouchDB for tests

In this part of the Testing Django series we will learn how to set up a dedicated CouchDB database for our automated Django tests. It builds on the previous parts that can be found here:

Django supports different SQL systems natively. If you want to use NoSQL solutions for making your data persistent you need to tinker a little bit yourself sometimes. I used CouchDB and Couchdbkit in a Django project. One problem was that the test data was saved in the same CouchDB database as data that was entered through the “productive” web interface. Here I describe how to have separate a database for each case.

So let us get into the described trouble by changing our model to be made persistent as CouchDB document.

First we install Couchdbkit:

$ sudo easy_install -U Couchdbkit

Then we have to modify the settings.py so that our Django project is aware of it:

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'couchdbkit.ext.django', # Add this
    'fruitsalad.fruits', # Added in previous parts
    'django_nose', # Added in previous parts
    'lettuce.django', # Added in previous parts
)

And we have to specify (also in settings.py) a database to use. The database does not have to exist as Couchdbkit will take care of this.

COUCHDB_DATABASES = (
     ('fruitsalad.fruits', 'http://127.0.0.1:5984/fruits')
)

Then we adapt our model:

from couchdbkit.ext.django.schema import *

class Fruit(Document):
    name = StringProperty()
    color = StringProperty()

    def set_name(self, name):
        self.name = name

    def set_color(self, color):
        self.color = color

    def is_yummy(self):
        return(True)

    def become_brown(self):
        self.color = "brown"

    def disappear(self):
        self.color = "transparent"

Okay, let’s use Django‘s shell to create such a fruit object and save it to the CouchDB database.

$ ./manage.py shell
Python 2.6.5  
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from fruitsalad.fruits.models import Fruit

In [2]: f = Fruit()

In [3]: f.name = "Banana"

In [4]: f.color = "yellow"

In [5]: f.save()

We can check the existence by using curl:

$ curl -X GET http://127.0.0.1:5984/fruits/_all_docs
{"total_rows":1,"offset":0,"rows":[
{"id":"4292816409d02b99378f31dd79e9b347","key":"4292816409d02b99378f31dd79e9b347","value":{"rev":"1-88a1fdab4ca60892f3143ba09cdf6205"}}
]}

This means we have (as expected) one document in the database. To have a closer look at it:

$ curl -X GET http://127.0.0.1:5984/fruits/4292816409d02b99378f31dd79e9b347
{"_id":"4292816409d02b99378f31dd79e9b347","_rev":"1-88a1fdab4ca60892f3143ba09cdf6205","color":"yellow","doc_type":"Fruit","name":"Banana"}

Alternatively use CouchDB‘s dashboard Futon that might be accessible at

http://localhost:5984/_utils/database.html?orange_test
depending on your CouchDB configuration.

We have to adapt our model test a little bit for the new constellation:

class TestFruit(object):

    def setup(self):
        self.fruit = Fruit()
        self.fruit.set_name("Papaya")
        self.fruit.set_color("orange")
        self.fruit.save()

    def test_color(self):
        nt.assert_equal(self.fruit.name, "Papaya")
        nt.assert_equal(self.fruit.color, "orange")

    def test_yumminess(self):
        nt.assert_true(self.fruit.is_yummy())

    def test_color_change(self):
        self.fruit.become_brown()
        self.fruit.save()
        nt.assert_not_equal(self.fruit.color, "orange")
        nt.assert_equal(self.fruit.color, "brown")

    def teardown(self):
        self.fruit.disappear()
        self.fruit.save()

If we run the test now …

$ ./manage.py test
Creating test database 'default'...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
sync `fruitsalad.fruits` in CouchDB
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
No fixtures found.
nosetests --verbosity 1
......
----------------------------------------------------------------------
Ran 6 tests in 0.352s

OK
Destroying test database 'default'...

… we can see that the documents produced by it are tainting the “productive” database:

$ curl -X GET http://127.0.0.1:5984/fruits/_all_docs
{"total_rows":4,"offset":0,"rows":[
{"id":"4292816409d02b99378f31dd79e9b347","key":"4292816409d02b99378f31dd79e9b347","value":{"rev":"1-88a1fdab4ca60892f3143ba09cdf6205"}},
{"id":"c85742c0ea99bd21549a1f058e4f10e6","key":"c85742c0ea99bd21549a1f058e4f10e6","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"de4e8d3ef236bc243f3768aa7b476ffc","key":"de4e8d3ef236bc243f3768aa7b476ffc","value":{"rev":"3-b227881c2cda9384dd26934854e31347"}},
{"id":"f8f15a0be7715ae852e75379f8d0b115","key":"f8f15a0be7715ae852e75379f8d0b115","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}}
]}

With every run it will become more documents. But we can circumvent this problem easily. We simply modify settings.py so that the application uses a different database if it is invoked by the test runner.

if "test" in sys.argv:
    COUCHDB_DATABASES = (
	('fruitsalad.fruits', 'http://127.0.0.1:5984/fruits_test'),
        )

If we run the tests againg …

$ ./manage.py test

we see that the database fruits_test is filled instead of the fruits database.

$ curl -X GET http://127.0.0.1:5984/fruits_test/_all_docs
{"total_rows":12,"offset":0,"rows":[
{"id":"008a039053c555d05a16ecf3aac1f9d0","key":"008a039053c555d05a16ecf3aac1f9d0","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"07dcba342255cec410c78f5252dde9c3","key":"07dcba342255cec410c78f5252dde9c3","value":{"rev":"3-b227881c2cda9384dd26934854e31347"}},
{"id":"19fe1c9088dce91d618cc763675c4f66","key":"19fe1c9088dce91d618cc763675c4f66","value":{"rev":"3-b227881c2cda9384dd26934854e31347"}},
{"id":"3feb80ae5aa35730a987cf90e1de34a0","key":"3feb80ae5aa35730a987cf90e1de34a0","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"456bc0ce470e5d333122bcc9d67b4aae","key":"456bc0ce470e5d333122bcc9d67b4aae","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"6028d26a7bf1b6ee23c43feda9341680","key":"6028d26a7bf1b6ee23c43feda9341680","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"69504e487a73a114757f55f256b4597b","key":"69504e487a73a114757f55f256b4597b","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"86f5a61dee535ed60374ffa4426abcc9","key":"86f5a61dee535ed60374ffa4426abcc9","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"8d620c6b6399e94233c06034506c4779","key":"8d620c6b6399e94233c06034506c4779","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}},
{"id":"96317d87c5f1f27ad68dfb702458c79d","key":"96317d87c5f1f27ad68dfb702458c79d","value":{"rev":"3-b227881c2cda9384dd26934854e31347"}},
{"id":"a8ae90da0b0e60adac137fb92a16883f","key":"a8ae90da0b0e60adac137fb92a16883f","value":{"rev":"3-b227881c2cda9384dd26934854e31347"}},
{"id":"e9f63c6460d8fad804f174ff850869ce","key":"e9f63c6460d8fad804f174ff850869ce","value":{"rev":"2-62250a0970ba7b627322c0a3ef889f09"}}
]}

The only downer is that our test database is getting bigger and bigger. So we have to clean it. To that we modify
fruits/tests.py slightly. In the top of the file we add

import fruitsalad.settings as settings
from couchdbkit.client import Database

and then we chance the setup method of the testing class :

    def setup(self):
        # cleaning the database
        db_url = settings.COUCHDB_DATABASES[0][1]  
        database = Database(db_url)
        database.flush()
        # nothing new
        self.fruit = Fruit()
        self.fruit.set_name("Papaya")
        self.fruit.set_color("orange")
        self.fruit.save()

Now before every test run the previous database content is removed. Alternatively move this cleaning part into the teardown method. The advantage of the here presented solution is that you can have a look a the data in the database if you need to.

Testing Django – part 3 – tdaemon

In this part we have a short look at the tool tdaemon and it’s application for testing Django.

Check out the other parts of this Testing Django series:

Writing tests in one thing – running them frequently is another thing. The lazy coder uses an change induced, automated test runner for this – this is not yet proper continuous integration testing but a first, local step into that direction. A good option for Django (and Python in general) is tdaemon.

The installation is pretty simple. Download the tdaemon.py from github, make the file executable and put it in your ~/bin folder (alternative just put it the project folder and call it with it’s path).

Once you have set it up you can run it in your Django project folder in the following manner:.

$ tdaemon --test-program=django

Whenever a file is changed tdaemon will detect that and automatically run the tests for you. Isn’t that handy?

Testing Django – part 2 – lettuce

The first part of this series was an introduction to Django testing with nose. In this part we will shed light on lettuce to practice Behaviour Driven Development (BDD). You can find all the other parts here:

The frame work lettuce is heavily inspired by cucumber – a BDD framework for Ruby. The key idea is to make specifications a communication tool between the different stake holders of a project (e.g. owner and the developer) and testable as well. To accomplish that the feature descriptions are written in a business-readable domain-specific language. A second layer (the so called step definition) is used to translate these specifications into testable code.

Starting with the dummy project from the first part we first need to install lettuce:

$ sudo pip install lettuce

And we have to configure the app to make use of lettuce (in the settings.py file):

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'fruitsalad.fruits', # From the first part
    'django_nose', # From the first part
    'lettuce.django' # Add now
)

By doing so manage.py gets a new command called harvest. Running this command will give the following output:

$ ./manage.py harvest
Django's builtin server is running at 0.0.0.0:8001
Oops!
could not find features at ./fruits/features

0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)

This means we have to add features. So we setup a dedicated folder

$ mkdir fruits/features

and create a file called fruits/features/fruits_basics.feature. The precise name is not important as long as it ends with .feature. Let’s add the a feature definition to this file:

Feature: Fruits should be yummy.

    Scenario: Having a cherry
        Given I access the url "/fruits/cherry"
        Then I see the content of the cherry page

Lettuce digests this feature and looks for the step definition of it. As there is none so far it offers us some snippets that we can use:

$ ./manage.py harvest
Django's builtin server is running at 0.0.0.0:8000

Feature: Fruits should be yummy.              # fruits/features/fruits_basics.feature:1

  Scenario: Having a cherry                   # fruits/features/fruits_basics.feature:3
    Given I access the url "/fruits/cherry"   # fruits/features/fruits_basics.feature:4
    Then I see the content of the cherry page # fruits/features/fruits_basics.feature:5

1 feature (0 passed)
1 scenario (0 passed)
2 steps (2 undefined, 0 passed)

You can implement step definitions for undefined steps with these snippets:

# -*- coding: utf-8 -*-
from lettuce import step

@step(u'Given I access the url "(.*)"')
def given_i_access_the_url_group1(step, group1):
    pass
@step(u'Then I see the content of the cherry page')
def then_i_see_the_content_of_the_cherry_page(step):
    pass

0 feature (0 passed)
0 scenario (0 passed)
0 step (0 passed)
----------------------------------------

We copy the given code into a the step file fruits/features/fruits_basics.py and run lettuce again:

$ ./manage.py harvest
Django's builtin server is running at 0.0.0.0:8000

Feature: Fruits should be yummy.              # fruits/features/fruits_basics.feature:1

  Scenario: Having a cherry                   # fruits/features/fruits_basics.feature:3
    Given I access the url "/fruits/cherry"   # fruits/features/fruits_basics.py:5
    Then I see the content of the cherry page # fruits/features/fruits_basics.py:9

1 feature (1 passed)
1 scenario (1 passed)
2 steps (2 passed)
----------------------------------------

The test passes now but only as we did not assert anything so far. Let change this. We use Django‘s testing client which we covered shortly in the first part to test the existence of a page:

                                                            
from lettuce import step
from lettuce.django import django_url
from lettuce import world
from django.test.client import Client

@step(u'Given I access the url "(.*)"')
def given_i_access_the_url_group1(step, url):
    world.client = Client()
    world.response = world.client.get(django_url(url))

@step(u'Then I see the content of the cherry page')
def then_i_see_the_content_of_the_cherry_page(step):
    assert world.response.content == "Show the cherry page"

world is a little helper construction that is explained here. Let’s run lettuce again with these real specs:

/manage.py harvest
Django's builtin server is running at 0.0.0.0:8000

Feature: Fruits should be yummy.              # fruits/features/fruits_basics.feature:1

  Scenario: Having a cherry                   # fruits/features/fruits_basics.feature:3
    Given I access the url "/fruits/cherry"   # fruits/features/fruits_basics.py:8
    Then I see the content of the cherry page # fruits/features/fruits_basics.py:13

1 feature (1 passed)
1 scenario (1 passed)
2 steps (2 passed)

The test passes and we are happy. We only applied Python‘s assert function in these examples but as described here nose’s assertion (e.g. assert_equal, assert_true) functions that we met in the first part of the series can be used here, too.

Konrad's Kode, Konfigs & Konstructions