Upgrading pywws on a pi

Today I am upgrading the weather station data logging computer's software. It's going sloooooowly. But once done it should allow better data analysis. Here's what's going on...

Today I am upgrading the weather data logging system I use to gather and process readings from my anemometer, temperature and humidity sensor, rain gauge, and barometer. Specifically I am upgrading the software used by the computer that logs and processes the data, pywws. The upgrade is taking some time.

The reason for the upgrade is that I want to be able to export weather data to an InfluxDB time series database to allow processing alongside air quality data. I hope to use a package like Grafana to compare and contrast data from different sources in one place.

The computer concerned is a Raspberry Pi Model B version 1, which explains why the upgrade is not moving especially rapidly. I have upgraded the system software (Raspian, based on Debian) to Stretch, ie. version 9. Buster (version 10) is available, but I can only bear so much waiting for an apt progress bar to creep its way across the terminal window. Buster will have to wait.

The really slow thing, though, is the upgrade of pywws, the actual logging package. It is an excellent package and has been rock solid stable for me for years. So stable in fact that I was both amazed and embarrassed to discover I was still running a version from 2013! The last time I upgraded it I used the tarball method, but it seems that is no longer de rigueur and pip is now the way to do things. Pip also seems to offer the promise of relatively painless upgrades. We shall see about that, but it certainly makes managing dependencies more straight forward, a la apt.

Pip, more accurately pip3 because we are using Python 3 now, however, is not fast on a version 1 Raspberry Pi. It takes a while to start up each time before deciding what to do and then getting in with it. This made a few false starts very time-expensive, but I think we are there now in terms of software installs.

It is late afternoon at the time of writing but we are still not done with the upgrade, because with the pywws upgrade also comes the need to reprocess the raw data gathered by the weather station over the years. It has been running now for nearly 8 years, and we are currently reprocessing data from 2016. The raw data itself is not changed, but the way pywws summarises the data for itself to allow tables and graphs to be produced (eg. those that go up on the website) does change from version to version. This 19.7.1 version doesn't like the way the older 2013 version used floating point values for some fields, and so refuses to do any calculations until the reprocessing is complete.

Hopefully once it is complete I can fire up the pywws-livelog program again and we will be live once more. Watch this space.

Update 07:57, 14 Nov: pywws is logging data again, but it has some issues with the old templates and weather.ini settings. Investigating.

Update 10:21, 14 Nov: Now producing weather graphs, but the text summaries are still causing trouble with tuples.

Update: 22:21, 14 Nov: Text templates are now processing just fine. Problem seemed to be the 'x' variable used in templates was being used in calculations, and pywws doesn't like that anymore. Problem fixed by using built-in special functions to calculate the required values, rather than hand crafted calculations (e.g. for wind speed in mph, or wind direction).

I think we can finally say, upgrade complete!