Tuesday, 21 November 2017

Concatenating CSVs the easy way

I was recently asked how to 'merge' a few CSV files into one, and if there was a script or tool that could do that.

Lets say you have 3 CSV files, called 1.csv, 2.csv and 3.csv
All you need to do is decide which CSV you want as the 'master' file (1.csv) then delete the header row from the rest.   Then simply do the following:

In Linux, Unix or OSX

cat 2.csv >> 1.csv
cat 3.csv >> 1.csv

In MSDOS, Windows or any other variant

type 2.csv >> 1.csv
type 3.csv >> 1.csv


>> on a command line means "Append to"

cat / type is simply a command to display the file

So cat 2.csv >> 1.csv is saying read 2.csv and append it to 1.csv.

Tuesday, 19 September 2017

Raspberry Pi Zero Smart Energy Meter


The plan for this project is to take my ageing Smart Energy meter that I got given free about 10 years ago, connecting it up to the very low-cost Raspberry Pi via a serial to USB interface, then using some Python scripts to 'webify' it and send the data to the home server to make graphs and stuff.

Not the world's most complex of hardware hacks as it just involves plugging things together but for a first dip back into the world of that kind of thing, I think it's enough.

Project overview:


Objectives: Getting the electricity usage data into a MySQL database on the server so that I can analyse it and build nice graphs and stuff and things.

Challenges: Parsing the stream of XML data, then firing it to the server using some kind of API endpoint (that I have to write)

Progress:


01/05/2016

  • Pi Zero ordered!


05/05/2016

  • Pi Zero delivered!


09/05/2016 - 18:20

  • I just successfully booted the Pi Zero, connected to the old USB hub. Not a major milestone, but a very important one.


It didn't explode! Yay! 




August 2016

  • We moved house and the Smart Energy meter has gone missing - it's in a box somewhere


September 2017

  • Still no further progress but I'm posting this project as it stands in an effort to kick me into actually making progress.  Hopefully there will be an update soon! 




Posting old stuff that I never finished - A "Weekly links" from sometime in 2014!

As part of an effort to post stuff that has been in draft for three years - here's this..   A "Weekly links" from sometime in 2014!

😜🔗🔗🔗🔗🔗🔗🔗🔗🔗


"Weekly"  !?!!

Anyway, moving swiftly along before anyone notices..

Some cool stuff I have found recently..

The story of the Amstrad CPC464 via The register

A Fish Tank on Wheels - drivable by the fish. No really.  via Hackaday


Monday, 4 September 2017

New theme

The old colour scheme was hurting my eyes.   Think I must be getting old.


Sunday, 3 September 2017

Raspberry Pi DVR - Streaming TV over the LAN


In our house we have a TV in an area where we don't have an aerial socket.  We hardly ever watch live TV any more anyway as we have an Amazon Firestick -  so it's not been that much of a problem.

Occasionally, however, we want to watch something that's live and no all the channels have 'Watch Live' as an option on their app. (Channel 4, I'm looking at you!)

So, last week I was reading the Pi Hit's blog and saw a short note about a DVR project that can stream live TV over the local LAN.   Just what we needed.


Friday, 21 April 2017

Oops - Time Flies when you're having fun..

Time flies..  or Chronosquittos?
So it's been a little while since the last post...

This, however, is a new start.   I'll leave all the old stuff because it's pretty good reference material still and most of the links still work.

I hope to be doing more stuff to put on here. Obviously I have done some bits in the last four years too so I'll add some of that as I go.

The main technology changes in the last four years, I would guess, are to cost and batteries.  You can get a Raspberry Pi Zero W for £10 which vastly out-powers any £1000 PC that I had 15 years ago, and has WiFi built in, and you can get batteries that recharge in hours that will power a heavy duty hammer drill or lawn mower.  Oh, and 3D printers.  Those are a big thing now.

Tidying up some loose ends from old posts - I can sync my iPod again! I managed to re-incarnate it to a level where it will sync with my work macbook pro, but I've stuck to my word and not tried to update the OS, so it's still running an iOS version from, like, 1400BC or something, but it's perfect for playing bed time stories to our son, so that's cool.






Monday, 16 September 2013

How to get text from a screenshot using OCR

An interesting conundrum today - how to get a plain text copy of a long list of options in a bit of software that isn't copy/paste-able.
OCR This! :-)

I should point out before I start that I'm using Ubuntu 12.04

So, anyway, I screen-captured the areas of the screen that formed the list into a series of image files. At this point I tried OCRing one of them straight off the bat, using the tesseract OCR.

Friday, 12 July 2013

Samsung Galaxy S3 Sensor Data

Recently I was fiddling around with a program on my S3 called Sensor Dump which just dumps the raw data from the various sensors onto the screen, and/or saves them into a CSV file.

Looking at the mag-field data on a graph is pretty interesting. This is where I passed my phone over the top of a laptop power supply.



It occurs to me that you could overlay this data on a camera display and do some kind of really low-resolution magnetic field camera thingie.

Hmm.   I wonder how directional it is.

Further tests required.

Thursday, 14 February 2013

Ah, Android. Always looking out for me.

So my phone, currently being used to play MP3s while I type documentation, just informed me that I should "consider reducing the volume to prevent damage to your ears" - well, thanks Android Music Player, that's very kind of you to say so. You're right. You're just 30 years too late.. (Damn you Not-Sony Not-quite-Walkman!!)

And also hugely underestimating the crapness of these £1 headphones - they don't go loud enough to damage my ears.

Monday, 28 January 2013

Experiments with really cheap hardware

Recently a new "pound shop" opened near my place of work.  Everything costs £1.
You'd be surprised how many people at the till still say "how much is that then?" 

Anyway, they sell all the usual pound shop stuff; sweets, plastic flowers, unknown brand batteries, blue powerade that no doubt makes your brain itch.

They also have an aisle that has random bits of tech.  So I thought I'd try some out. 

The first I tried was a micro bluetooth USB dongle. I took the lid off it to make sure it wasn't just going to short out and burn my PC to the ground - it looked ok, so I tried it on an old sacrificial PC..  and it worked just fine.. Not the fastest in the world, but functional. 

The next thing I tried was the headphones.. Not quite so successful as the cable is only about 750cm long, and my tower PC is on the floor - so I end up stooped over to hear anything. Also the plug seems to be a little bit too short, and you have to really wiggle it around to get both speakers working correctly.  

Next - A "Blakberry USB charger cable" - yes, that's how it was spelt.  It's actually a standard micro USB cable, I checked the pins using a multimeter and they appeared to all be connected as they should be. Tried it on a sacrificial PC again, and an old phone (paranoid much?!)   and yep, worked first time - full data connection.  Not bad for £1.   

They also do car phone chargers, watches, HDMI cables, SCART cables, and all manner of things with LEDs in. All good stuff for future projects.


Monday, 17 December 2012

Linux, Command line email and config errors.

I've been battling trying to send a command line email from an Ubuntu system.   It all looked like it was going, and the mails were being queued to send, but nothing ever actually went anywhere.

Checking the /var/log/mail.err file I see


 postfix/sendmail[23197]: fatal: open /etc/postfix/main.cf: No such file or directory

(and a bunch of other similar ones) 

One quick google later, and it turns out that, even though it was a clean install of the mail program, the config files didn't get written out properly. 

To fix it, I did 

sudo dpkg-reconfigure postfix

Which runs the setup that ran when mail was originally installed.   

The PostFix Configuration Wizard!
(check out the 1990s ANSI styling)

That seems to have fixed the problem, and all the mail that was queued came through. 

One to remember - so I thought I'd chuck it on t'blog. 


Friday, 16 November 2012

MySQL and vanishing InnoDB engines

So, apparently, MySQL has an amazing ability to make InnoDB engines vanish.

If that sentence means nothing to you, I apologise, but this is going to be an exceptionally geeky post, even for me.

If you use MySQL with InnoDB tables, it seems that they can very easily stop working because the InnoDB engine just goes away. It can be for a number of reasons, but they all seem to revolve around the "ib_logfile" files.

If, for whatever reason, you change the size of the InnoDB log in the config, and don't delete the existing log file - MySQL will start perfectly happily, but you won't be able to access anything InnoDBish.   Similarly if the ib_logfile(s) get corrupted in anyway.

Doing a "show engines \G" in the MySQL prompt will show you what's wrong - there is no InnoDB engine running.

You won't get any error messages. No warnings will be displayed. The only clue will be when you try to access the table and it will give you a "No InnoDB engine" error.

The solution is to delete or move the existing ib_logfiles from the data folder, restart the MySQL service and it will create new ones.  Everything should now be back to normal.


Apparently this can also happen if the log file setting is too high and the memory can't be allocated - so that's worth keeping an eye on.

Monday, 22 October 2012

Charge directly, not via hub..

Having an electronics background, I really should've realised this sooner - however - if you're using a USB data cable to charge your smartphone, and it seems to be taking forever - is it plugged into an unpowered, very cheap, USB hub?
Cheap USB Hub
"We need more power captain!"

Turns out mine was, and was taking about an hour for 1% of charge, because the power supplied by the hub is so little it was only just managing to supply more than the phone was using.

I plugged it directly into the USB port on the PC and it's gained 2% in 5 minutes.

Might be a useful tip, so figured I'd stick it on t'blog.

Monday, 25 June 2012

Raspberry Pi - ordered

I have ordered a Raspberry Pi.

It would seem I have "up to 10 weeks" to get my brain (and workshop) in gear to think what I'm going to do with it.

Lots of cool basic ideas.  None of them have actually solidified into anything resembling a "plan" yet.

----------

Update - 13/8/2012 - still no Pi.

I bid on one on eBay, but it eventually went for over £100, which considering it was a bog-standard one without even the O/S card, seems a bit crazy-insane.

---------

Update #2 - 23/8/12 - Woohoo!!

"You have a shipment being sent from RS COMPONENTS LTD that is due to be despatched on 22/08/2012" Looks like it's on it's way!!

---------

Update #3 - 04/09/12 - ARRIVED

It arrived..   I haven't done anything with it yet as have been too busy, but I have lots of plans.

Wednesday, 25 April 2012

Angry birds and mouldy Apples

This is a story of how my continuing quest to get Angry Birds Space on my iPod led me through a journey of rage, disappointment and finally, acceptance that everything is, in fact, just a bit crap.

dead ipod
I finally bit the bullet and updated my iPod touch. You may remember to do it from my Mac would basically require me to buy a new Mac, so I installed iTunes 10.6 on my PC, upgraded the iPod and got it running iOS 4.2.1

I tried installing Angry Birds Space again - the original reason for the upgrade - only to be told that it doesn't support that particular iPod HARDWARE because it needs better graphics capabilities..

A small red light began to blink in my brain as my rage inhibitor burnt out.

"You could've told me that....  "    I said quietly...

The skies darkened, the crackle of rage filled the air and everything fell eerily quiet. 

"...BEFORE I UPGRADED THE O/S!!!"  

Tuesday, 10 April 2012

Parsing bookmark files using PHP and writing it out as an HTML link

Here's a code example of something I knocked together to import Windows .url shortcut files to a MySQL database on my home-server system that I'm building.

The shortcut files are actually just text, so you can read them really easily..  
They just contain a header, then the URL.. For example:

[InternetShortcut]
URL=http://www.hackerspace.co.uk


Here's a PHP code snippet... 


//  Create a function and give it the filename of the URL file.. 

function readurl($filename)
{

// Pull the .url off the end.. 
$filenamebits= explode(".url",$filename);

// Create a "Link Description" from the filename
$linkdesc=$filenamebits[0];

// Open the file to get at it's juicy contents
$file = fopen("$filename","r");

// Set up a counter
$linecount=0;

// Set up a "While there's still stuff in the file" loop
while (!feof($file) ) {

// Open the file and get stuff from it, line by line
$line_of_text = fgets($file);

// If you're on line number 1... 
if ($linecount==1)

// Explode the line of text into pieces
$pieces = explode("=", $line_of_text);

// Extract the bits we want and wrap it in some friendly HTML
$URLTOADD="<a href=\"$pieces[1]\">$linkdesc</a>";

// You can either then write this string out to the screen, or like I do, fire it 
// into a database for later use

}

// Increment the line count
$linecount++;

}

// Close the file. Otherwise the operating system gets cross. Probably. 
fclose($file);

}

Why I can't play Angry Birds Space.

I wanted to play Angry Birds Space.  It's a simple thing. Should've been a no-brainer.

I tried it on my iPod touch, but it doesn't have iOS 4, so I'm deemed not-worthy.  I can't upgrade to iOS 4 because to do that requires a newer version of iTunes than my Mac OS version will let me install.  I can't upgrade my Mac OS because it's an old Mac and won't RUN a newer version..   So, to play it on my iPod, I would need to buy a new Mac.
Not going to happen.

Or I could install iTunes on my Windows PC and struggle to sync my music over, and then struggle to get it to do anything because the Windows version of iTunes is just not very good. (Sorry)

So, I thought I'd try the Android version on my phone..  I went to the Google app store thing (or Google Play as it now appears to be called) and went to download it.  Glancing at the permissions made me stop in my tracks..

Thursday, 22 March 2012

Dropbox, iOS and Time related problem!

So, I just tried to get a screenshot of Angry Birds Space telling me I couldn't install it because I needed iOS 4 on my iPod touch. Which I can't do easily for long, complex, stabby making reasons..

Anyway..  Tried to upload it to dropbox and it continually failed..   Which was odd.

I ended up unlinking the dropbox app on my ipod, and then relinking it. Only then did it tell me that it couldn't connect because my system clock was more than 10 days out of date.  (Due to a battery charge issue. Well, me not charging the battery for about 2 months, anyway)

Reset the date, tried dropbox again, and hey presto it works. Worth remembering.

The picture I was trying to transfer. Damn you old iPod. 

Monday, 16 January 2012

Links of stuff and things

I've been pretty busy recently, so the weekly part of weekly links went out the window.   Still, here's some interesting stuff I've found recently..

HP give WebOS to OpenSource - Good plan. Could lead to some interesting alternative O/Ses for things.

Tron Derezzed music - played on floppy drives!

EZ-Robot - Upgrade your old robotic toys to something way more cool

How gaming can enhance real life - Very interesting article in New Scientist about how gaming reactions can spill into real life. It's pretty rare that an article like this doesn't pull in all the classic negative stereotypes.


Friday, 13 January 2012

Turn the beep off!

Random, quick tip - I can't believe I didn't know this already, but it turns out you can turn off the annoying DOS style BEEP from a command prompt in Windows.

Open a command prompt.

Type
net stop beep

you should get a response of
The Beep service was stopped successfully.

To turn it back on -
net start beep. 

To test it type "echo [ctrl G]"  - which will look like


echo ^G

on screen.


Concatenating CSVs the easy way

I was recently asked how to 'merge' a few CSV files into one, and if there was a script or tool that could do that. Lets say you h...