Concatenating CSVs the easy way

I was recently asked how to 'merge' a few CSV files into one, and if there was a script or tool that could do that.

Lets say you have 3 CSV files, called 1.csv, 2.csv and 3.csv
All you need to do is decide which CSV you want as the 'master' file (1.csv) then delete the header row from the rest.   Then simply do the following:

In Linux, Unix or OSX

cat 2.csv >> 1.csv
cat 3.csv >> 1.csv

In MSDOS, Windows or any other variant

type 2.csv >> 1.csv
type 3.csv >> 1.csv

>> on a command line means "Append to"

cat / type is simply a command to display the file

So cat 2.csv >> 1.csv is saying read 2.csv and append it to 1.csv.

Raspberry Pi Zero Smart Energy Meter

The plan for this project is to take my ageing Smart Energy meter that I got given free about 10 years ago, connecting it up to the very low-cost Raspberry Pi via a serial to USB interface, then using some Python scripts to 'webify' it and send the data to the home server to make graphs and stuff.

Not the world's most complex of hardware hacks as it just involves plugging things together but for a first dip back into the world of that kind of thing, I think it's enough.

Project overview:

Objectives: Getting the electricity usage data into a MySQL database on the server so that I can analyse it and build nice graphs and stuff and things.

Challenges: Parsing the stream of XML data, then firing it to the server using some kind of API endpoint (that I have to write)

Progress:


01/05/2016

• Pi Zero ordered!

05/05/2016

• Pi Zero delivered!

09/05/2016 - 18:20

• I just successfully booted the Pi Zero, connected to the old USB hub. Not a major milestone, but a very important one.

It didn't explode! Yay! 

August 2016

• We moved house and the Smart Energy meter has gone missing - it's in a box somewhere

September 2017

• Still no further progress but I'm posting this project as it stands in an effort to kick me into actually making progress.  Hopefully there will be an update soon! 

Posting old stuff that I never finished - A "Weekly links" from sometime in 2014!

As part of an effort to post stuff that has been in draft for three years - here's this..   A "Weekly links" from sometime in 2014!

😜🔗🔗🔗🔗🔗🔗🔗🔗🔗


"Weekly"  !?!!

Anyway, moving swiftly along before anyone notices..

Some cool stuff I have found recently..

The story of the Amstrad CPC464 via The register

A Fish Tank on Wheels - drivable by the fish. No really.  via Hackaday

New theme

The old colour scheme was hurting my eyes.   Think I must be getting old.

Raspberry Pi DVR - Streaming TV over the LAN

In our house we have a TV in an area where we don't have an aerial socket.  We hardly ever watch live TV any more anyway as we have an Amazon Firestick -  so it's not been that much of a problem.

Occasionally, however, we want to watch something that's live and no all the channels have 'Watch Live' as an option on their app. (Channel 4, I'm looking at you!)

So, last week I was reading the Pi Hit's blog and saw a short note about a DVR project that can stream live TV over the local LAN.   Just what we needed.

Oops - Time Flies when you're having fun..

Time flies..  or Chronosquittos?

So it's been a little while since the last post...

This, however, is a new start.   I'll leave all the old stuff because it's pretty good reference material still and most of the links still work.

I hope to be doing more stuff to put on here. Obviously I have done some bits in the last four years too so I'll add some of that as I go.

The main technology changes in the last four years, I would guess, are to cost and batteries.  You can get a Raspberry Pi Zero W for £10 which vastly out-powers any £1000 PC that I had 15 years ago, and has WiFi built in, and you can get batteries that recharge in hours that will power a heavy duty hammer drill or lawn mower.  Oh, and 3D printers.  Those are a big thing now.

Tidying up some loose ends from old posts - I can sync my iPod again! I managed to re-incarnate it to a level where it will sync with my work macbook pro, but I've stuck to my word and not tried to update the OS, so it's still running an iOS version from, like, 1400BC or something, but it's perfect for playing bed time stories to our son, so that's cool.

How to get text from a screenshot using OCR

An interesting conundrum today - how to get a plain text copy of a long list of options in a bit of software that isn't copy/paste-able.

OCR This! :-)

I should point out before I start that I'm using Ubuntu 12.04

So, anyway, I screen-captured the areas of the screen that formed the list into a series of image files. At this point I tried OCRing one of them straight off the bat, using the tesseract OCR.

Samsung Galaxy S3 Sensor Data

Recently I was fiddling around with a program on my S3 called Sensor Dump which just dumps the raw data from the various sensors onto the screen, and/or saves them into a CSV file.

Looking at the mag-field data on a graph is pretty interesting. This is where I passed my phone over the top of a laptop power supply.

It occurs to me that you could overlay this data on a camera display and do some kind of really low-resolution magnetic field camera thingie.

Hmm.   I wonder how directional it is.

Further tests required.

Ah, Android. Always looking out for me.

So my phone, currently being used to play MP3s while I type documentation, just informed me that I should "consider reducing the volume to prevent damage to your ears" - well, thanks Android Music Player, that's very kind of you to say so. You're right. You're just 30 years too late.. (Damn you Not-Sony Not-quite-Walkman!!)

And also hugely underestimating the crapness of these £1 headphones - they don't go loud enough to damage my ears.

Experiments with really cheap hardware

Recently a new "pound shop" opened near my place of work.  Everything costs £1.

You'd be surprised how many people at the till still say "how much is that then?" 

Anyway, they sell all the usual pound shop stuff; sweets, plastic flowers, unknown brand batteries, blue powerade that no doubt makes your brain itch.

They also have an aisle that has random bits of tech.  So I thought I'd try some out. 

The first I tried was a micro bluetooth USB dongle. I took the lid off it to make sure it wasn't just going to short out and burn my PC to the ground - it looked ok, so I tried it on an old sacrificial PC..  and it worked just fine.. Not the fastest in the world, but functional. 

The next thing I tried was the headphones.. Not quite so successful as the cable is only about 750cm long, and my tower PC is on the floor - so I end up stooped over to hear anything. Also the plug seems to be a little bit too short, and you have to really wiggle it around to get both speakers working correctly.  

Next - A "Blakberry USB charger cable" - yes, that's how it was spelt.  It's actually a standard micro USB cable, I checked the pins using a multimeter and they appeared to all be connected as they should be. Tried it on a sacrificial PC again, and an old phone (paranoid much?!)   and yep, worked first time - full data connection.  Not bad for £1.   

They also do car phone chargers, watches, HDMI cables, SCART cables, and all manner of things with LEDs in. All good stuff for future projects.

Linux, Command line email and config errors.

I've been battling trying to send a command line email from an Ubuntu system.   It all looked like it was going, and the mails were being queued to send, but nothing ever actually went anywhere.

Checking the /var/log/mail.err file I see


 postfix/sendmail[23197]: fatal: open /etc/postfix/main.cf: No such file or directory

(and a bunch of other similar ones) 

One quick google later, and it turns out that, even though it was a clean install of the mail program, the config files didn't get written out properly. 

To fix it, I did 

sudo dpkg-reconfigure postfix

Which runs the setup that ran when mail was originally installed.   

The PostFix Configuration Wizard!
(check out the 1990s ANSI styling)

That seems to have fixed the problem, and all the mail that was queued came through. 

One to remember - so I thought I'd chuck it on t'blog.