Vim security

Just read https://glyph.twistedmatrix.com/2015/11/editor-malware.html and https://jordaneldredge.com/blog/why-i-switched-from-vundle-to-plug/ which both discuss some issues about text editor security.

TL,DR:

1. vim-plug is preferred to vundle
2. developers may be seen as high value targets for system compromise.
3. beware git:// protocol

A Python Christmas present

Nose is my current Python testing framework of choice.

When I'm doing TDD which is more or less all the time*, I'm capturing logs and (shock horror) print statements and I commonly invoke nosetests in a very noisy way with the -sv switches.

A few days ago, I found myself thinking about writing an output colourizer for nose when (gasp) it occurred to me that one might already exists. And, wonder of wonders, I was right.

I would like to introduce you to rednose. It's nearly Christmas, the plugin (appropriately) is called 'rednose' and it represents the gift of visually interesting test results.

Rednose is easy to install, easy to invoke, and it works like a dream. I was especially pleased with it's ability to colourize logs and stack traces.

Kudos to the developer https://github.com/timbertson.

* If I'm not doing TDD I'm researching an unfamiliar API inside an IPython/Jupyter notebook.

Jupyter notebooks

My typical working setup is a headless GNU/Linux VM which I access via ssh. This means that when I working in a newly initialized project environment and I want to pop up a Jupyter notebook I do this:

pip install jupyter
jupyter notebook

And bang! there I have a w3m text based browser, nicely colourized, but not what I was hoping for.

Every month or so I go through this. And I have been trained to find the Jupyter config file or the IPython config file as was and edit it to suit.

So this time the version of Jupyter which pip harvests for me is 4.0.6. And could I find the configuration? No I could not.

It turns out you have to generate the configuration before you can actually edit it:

jupyter notebook --generate-config

Once that is done, then you can

1. change the host from localhost to '0.0.0.0' and
2. change start browser by default to 'False'.

Oh, and look for the configuration in

.jupyter

.

Probability in business

As someone who has put together quite a few business plans over the years, this short (4 minute) video from Prof Sam Savage struck me as quite insightful.

For me it highlights the difference between those of us with a knowledge of the theory of probability and those who have really thought about the way probability matters manifest in daily life. (I'm somewhere in between, and probably (hmm interesting word choice) closer to the theoretical end).

I notice that in the background, Prof Savage has a copy of Bernstein's "Against the Gods" which I am pleased also to own and to have read, although in any discussion on probability, it would be a crime not to mention "Fooled By Randomness" from Nassim Nicholas Taleb.

And now that I'm thinking of statistical solecisms, the common theme between Savage, Bernstein and Taleb, I'm reminded of the joke about the goverment education minister who was often heard to complain of the large number of schools which were still below average.

Enjoy the video.

Using Python unittest outside a testing environment

Test frameworks are not just for testing code during development and deployment.

Test frameworks may also be put to good use within your application.

By way of example, consider the possibility of running diagnostics on a live application. Another possibility is checking the validity of an on-the-fly configuration change.

The Python unittest library, part of the standard distribution can easily be incorporated into your live code.

The code example below presents a toy example of a test case, which is then exercised by your application process. The outcome is a test result object which you can query and report on.

import unittest


class TestMe(unittest.TestCase):
    def test_this(self):
        self.assertTrue(False)
    def test_that(self):
        self.assertTrue(True)

loader = unittest.TestLoader()
suite = loader.loadTestsFromTestCase(TestMe)
result = unittest.TestResult()
suite.run(result) # run actually returns result as well as populating it
print result

<unittest .result.testresult="" errors="0" failures="1" run="2">

The variable 'result' is an instance of unittest.TestResult() and has a rich set of attributes and methods as documented here.

Python3 Namespace Packages Using Wheels

I was experimenting with Python3.4 and namespace packages.

Here's what I did.

I created two completely separate repos, A and B.
I had a top level namespace, toplevel.

I created a directory structure:

repoA/toplevel/libA/moda.py
repoA/setup.py

repoB/toplevel/libB/modb.py
repoB/setup.py

The setup.py files were very similar:

from setuptools import setup

setup(
    name = "toplevel libA",
    version = "0.1",
    packages = ['toplevel.libA']
)

and:

from setuptools import setup

setup(
    name = "toplevel libB",
    version = "0.1",
    packages = ['toplevel.libB']
)

To be able to work with the bdist_wheel extension to setuptools, I had to use pip to install wheel.

pip install wheel

Then in repoA and repoB it was possible to execute:

python setup.py bdist_wheel -d /tmp/wheels

In this way, two wheels were created:

/tmp/wheels/toplevel.libB-0.1-py3-none-any.whl
/tmp/wheels/toplevel.libA-0.1py3-none-any.whl

I then created a third project repo which used these two wheel packages.

repoC/someotherproject

In this repo, after creating and virtual envirionment, it was possible to install the two wheels:

pip install /tmp/wheels/toplevel.libB-0.1-py3-none-any.whl
pip install /tmp/wheels/toplevel.libA-0.1py3-none-any.whl

In the site-packages directory of the repoC environment there was a tree structure installed:

./toplevel/liba
          /libb

Showing that libA and libB, although separately developed, were available in the client project as a unified namespace. (Note that liba and libb could have been independently installed on different branches of sys.path and they would stilll appear to the interpreter as an integrated whole.)

Getting started with Sphinx

I really like pdoc as a great lightweight python source code documentation tool.

I like to invoke it:

PYTHONPATH=. pdoc --http --http-host 0.0.0.0 --http-port 8888 --only-pypath

I like to use ReST to format docstrings. Unfortunately I have not been able to get pdoc to display parameter lists as I would hope.

(UPDATE: 20151205: pdoc will honour parameter lists created using Markdown.)

So I switched to using Sphinx. It seems I am not alone in finding the Sphinx documentation hard to use.

After a bit of futzing around, this recipe meets my objective of having nice documentation with the power of Sphinx but the ease of pdoc.

1. Install Sphinx.
2. Invoke sphinx-apidoc -F -o dox
3. cd to the dox directory.
4. Edit conf.py to ensure that sys.path can find the module to be document.
5. make html
6. python3 -m http.server

I hope it works for you.

Scrum and Working Code

So I bought this machine for the production line.

Who'd you get it from?

Acme. I told them what I needed, they would build a bit of it, show it to me and so it went, round and round until I got the machine installed on the line. Works well. Or rather it did.

It's not working?

No it works. It works just the way it always did. None of the parts wear out. It's a marvel really. But we got the new WhizzBang mark II six months ago. It sits in the line just in front of the Acme. Beautiful piece of kit. Spits out widgets a whole lot faster than the previous mark I model.

And the Acme can't keep up?

Yup. You got it. Never thought the WhizzBang people would ever be able to make faster machine and the new mark II is a sight to see. But I never thought to ask the Acme people to put a speed knob on the control panel.

Ah.

Ah indeed. But I'm a resourceful guy. I thought maybe there was some other way to speed it up. Use a lighter lubricant, tighten up some spring tensioners, shorten the belt, or perhaps up the voltage a bit, maybe upgrade the motor. I asked Fred in maintenance to bring me up the manual for the Acme.

And Fred had lost the manual?

Close. Fred says he never got the manual. My fault really because I was so focussed on getting the Acme built, I guess asking for a manual just got overlooked. Anyway I got on the phone to Acme and asked them to send me one over. They said they would get right on it.

But they never did?

Right again. Apparently when they went looking, there was a binder for the manual but there was nothing in it, just a note to ask Engineering. Apparently because our machine was a one-off, and I didn't specifically ask for a manual, nobody thought to make one. Apparently that's the way things are these days.

You didn't give up that easily did you? Did you?

Oh no. I'm a get-things-done kind of a guy. I got right back at them. I said OK, send me over the engineering drawings and I'll work out how this machine works myself. It might take me a while, but I can read a good set of plans pretty well. Yeah - I'll save you the bother. There were no plans.

Good grief.

You won't believe how these guys worked over there at Acme. They tested everything - scales and calipers and stopwatches - you name it, they had it. They still had a test rig for the main motor assembly and their test results. That baby clocked 2500 rpm with a variance of less than 0.001%. I'll have to hand it to them, their tests looked pretty good. Somebody over there said every single part was tested. It's just that they haven't got a single piece of paper to tell me how the thing fits together.

Didn't you ask them how they built it without plans?

I think they were getting a bit defensive. I didn't have to ask them. The guy in charge - a pretty smart guy - knew what I was thinking so he just started explaining how their system worked. Apparently they would build test rig, decide what answer they wanted and then build the part until it passed the test. Over and over again for every different part. And when they started bolting parts together, they would build a test rig for the assembly. They carried on like that until they had the whole thing built. And yes, they even had a test rig for the whole machine. I guess maybe that's part of the reason why the parts haven't worn out. He seemed pretty pleased about that.

So what happened when you told them about your need for a speed control on the Acme you've got?

The best they could come up with was to say that they would get Engineering to come up with some suggestions. Apparently some of the team who built my Acme are still there. But since I put my order in two years ago, a lot of these guys have moved on. Apparently two years is a long time in their line of work. When I asked them how they were going to do it without any documents they said that the machine itself was the documentation.

So a sort of self-documenting machine?

Kinda.  And to be fair, many of the parts have names etched into them like 'big_sprocket_for_main_drive' and 'main_drive_axle_locking_assembly'. But it's like that physicist Feynman says, don't mistake knowing the name of something as being the same as understanding what it actually does.

Is there any good news? You said these were intelligent people.

Yes. They didn't need to disassemble my working Acme because they managed to find a perfect copy of all the components in their stock control system.

Well, that doesn't sound so bad. How is it working out?

Not so good. The few remaining engineers who worked on my project are all more senior now and have been allocated to new clients. Because trying to guess how the machine works staring at a bench full of parts is pain in the neck, nobody has that much enthusiasm. It's certainly taking a long time.


And the WhizzBang?

Well the money I spent on the upgrade has been wasted. Nothing can go faster than the Acme.

Would you do anything differently in the future?

I always thought that if I needed any changes I could go back to the maker. But when even the makers are unable to work out how they made the thing in the first place ... There's a lesson in there somewhere.

Python Logging

Just pulling together the best of the web on Python logging:

https://docs.python.org/2/library/logging.html
The standard library documentation.

http://pieces.openpolitics.com/2012/04/python-logging-best-practices/
Insightful and reviewed by Vinay Sajip.

http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python
Helpful tips on which logger levels to use.

http://stackoverflow.com/questions/15727420/using-python-logging-in-multiple-modules
Answer provided by Vinay Sajip.

Android Studio up and running

My standard development environment for native Android development has been Vim and Ant.

Today I finally took the plunge and got started with Android Studio (AS).

I ported in ZipZipBooks, which is my mobile record keeping application for HMRC with a view to giving at a bit of a refresh. Didn't take too long to get it up and running on AS.

Key issues:

1. The code for the existing project is in a Mercrurial repository. My first attempt to import the project, I went down the "Check out the project from version control" route, and chose Mercurial. Unfortunately this did not play niceley with Gradle. Instead, I found that I could instead start with the "Import Project" route. That allowed AS to convert my Ant build to Gradle.
2. Adding library dependencies for which this answer was very helpful;
3. I might need to get a faster/bigger/hotter/noisier laptop.
4. Running on Windows 8, there was no need to update any drivers for the physical Android devices on which I performed testing.

Reading vs Doing

I was greatly enjoying reading this article on coding when I got to the section on Django and decided that maybe I should get back to my own Django project.

Django High Level Mind Map

Mind maps really suit my way of organizing knowledge.

Here is a link to pdf of my high-level Django mind map.

Django High Level Mind Map

Enjoy.

The Monty Hall Problem

The Monty Hall problem is great fun.

Here's how I rationalise it.

There are three doors. I have to choose one. The chance that the door I pick is the correct one is one in three. The chance that one of the other doors is correct is two in three.

Stop right there.

Now ask yourself the question - is Monty Hall running around behind the doors possibly changing the location of the prize? Nope. The prize doesn't move. It's exactly behind the same door as it was when the puzzle started.

So no matter what happens, my chances of winning with the door I first chose was and remains one in three.

And remember that means there is a two in three chance that the prize is behind one or other of the doors I didn't choose.

Which means that before he opens one of the two doors I didn't choose it's 50/50 which door has the two in three chance of being correct. After Monty opens one of the doors that 50/50 uncertainty disappears.

Which means the two in three chance of being correct now applies to the one remaining closed door I didn't originally choose.

Since a two in three chance of being right is better than a one in three chance of being right, I should change my initial decision.

Programming is bad for your health

I've just been reading this article on why sitting still is bad for you and, if I read it right, standing still is not much better .

It is hardly news. We were designed to be active. But it is always uncomfortable to have the abstract awareness of harm condensed to actual damage vectors.

I've tried for years to get going the Pomodoro principle - but my concentration span doesn't seem to like the short 25 minute periodicity.

Time to make more effort perhaps.

GitHub Profile

Because a number of third party services seem to believe that you cannot possibly be a serious programmer unless you have an account on GitHub, I have had to create one.

But my shiny new GitHub profile is only a cypher to direct anyone who passes by to my public repositories on Bitbucket .