Installing Debian Lenny on ML370 G3 with RAID Controller.

Whilst Debian Lenny is compatible with the Compaq/HP ML370 G3 with RAID Controller on-board, it took me a little work to get it installed and I wanted to document my efforts for myself and others!  This was a painful process to get it working, and perhaps isn’t for the weak of heart or those early in their Linux-ninja training, but hopefully this will guide you to the path of enlightenment and joy!

PreReqs:-
An ML370 G3 with sufficient hardware to boot/install e.t.c.
A copy of HP SmartStart (I used 7.20, others may work). ftp://ftp.hp.com/pub/products/servers/supportsoftware/ZIP/
A copy of Windows 2003 (2000 may also work).
Debian Lenny NetInst CD
USB Stick formatted as FAT with ‘non-free’ Array Controller Driver saved to it (ql2300_fw.bin in my case).

How I did it:-

1. Connect up the machine, power on and insert the HP SmartStart CD
2. Once the SmartStart has booted, configure your array(s) as per your requirement.  In my case, I used 2x147GB drives in a RAID1 and 4x72GB drives in a RAID5.
3. Do an operating system install, choose to install Windows and let it do its own partitioning.
4. Let Windows install, don’t bother doing any major configuratoring as you’re going to wipe the server again shortly…
5. Once Windows is installed and on the desktop, insert the Debian CD and Reboot.
6. Boot to the Lenny Installer, and choose a ‘Standard Install’.
7. Configure as normal.
8. When you get to the bit about partitioning, choose the Manual Option.
9. Goto the NTFS partition that the Windows install created, change it to an Ext3 format, ensure it is set to be bootable, and set the mount point to be /.
10. Id leave other partition configuration until the point that you can boot Linux.
11. Carry on and install whatever packages/components you require as standard.
12. When you’re prompted to install the GRUB bootloader into the MBR – SAY NO
13. You’ll be asked where you want to install the bootloader.  My partition was /dev/cciss/c0d0p1.
14. Finish off the Installer, remove the CD and Reboot.
15. Fingers x’d, you’ll get GRUB boot menu.  Mine then refused to boot the OS saying that the Operating System wasn’t found.   I edited the initial boot menu because it was referring to HD (1,0).  I changed this to HD (0,0) and it booted.
16. Login to the console.
17. If the GRUB editing worked, make sure to edit /boot/grub/menu.lst and permanently change the setting to HD (0,0) for each of the menu items.
    

Cool Runnings:-
This server when running at full tilt with the fans is a noisy noisy beasty! However, you can control those fans to make them run at a dull roar rather than full hurricane force chat.  Follow the instructions from Jonas Bjork related to Ubuntu. Debian works exactly the same way, but you’ll need to install SNMPD and libstdc++2.10-glibc2.2 (the latter you’ll have to manually download the .deb from the Etch repository).  Once its all installed, run the ‘hpasm activate’ command and follow the on-screen instructions.

Profit!

Sky EPG –> Dreambox Bouquets.

I’ve recently updated the software on the Dreambox and wanted to put an updated Bouquet on there with all the channels lined up in the ‘sky’ style.  With over 1000 channels to sort through, I thought there had to be an easier way.  Sonic1 over on this forum HERE has put up a massive sorted list of satellite feeds all over Europe and I recommend you use it.  However, if you just want the Sky UK EPG layout only as your bouquets, I’ve reworked it and published it HERE . 

This is compatible with DreamBoxEdit (v3.0.0 tested) and you will need to update your preferred local BBC and ITV regions as appropriate.  Also, you may need to update the bouquet frequencies and listings if necessary, as they do get changed fairly often.

gOS(h)

As you may have heard, Google have decided to step into the Operating System arena by releasing one aimed at revolutionising the desktop OS market.  At launch, it will be targeted at the lightweight ‘netbook’ range of devices specifically for people who require regular, rapid access to the Internet.   Information coming from the Mountain View chocolate factory is that it uses the Linux kernel using a ‘new windowing system’, presumably their own GUI rather than relying on X Windows and KDE or Gnome.  Presumably it is a rework of the Android platform to better support larger screens, keyboards and mice, although they do say that the two projects are not intertwined.  The OS also has Chrome Browser built in and is designed to be an interface into ‘cloud’ computing resources that have been talked about over the last couple of years.  Because the big G now provides ‘office’ resources such as E-Mail, Documents, Spreadsheets and Presentations online, they don’t see the point in having massive computing resources on the desk to write a simple document, when their server infrastructure can provide that for free (on the basis that they’ll sell advertising to you) provided you can get online.

Its an interesting approach to take, and it will be interesting to see how it pans out.  However, I do have a few concerns.  Firstly,  the Internet is NOT yet all pervasive.  Sure, you might have it wall to wall at home, and you may have access in the office.  You may even have 3G on your phone or via a dongle to get online whilst you’re mobile,  but you don’t have to travel very far to enter an internet black hole.  For example in our Office,  I can get HSDPA access with T-Mobile and 3 Internet,  but BT/Vodafone barely even has a 2G signal.  Likewise driving home,  even though I travel the M6 corridor, one of the busiest routes on the motorway network,  there are at least 3 spots where the signal drops out completely.  At the moment, this isn’t a problem, because you don’t need access to the net 100% of the time to write a document or update your e-mail.  But if you’re sat there with a blank screen because your network has dropped out, frustration levels soon start to set in.   

Of course, you can get round all this with offline synchronisation, which is the one thing currently missing from another cloud computing environment you may be exposed to – Citrix.  However,  you’re then NOT working in the cloud when you’re working offline, which means more storage, processing and memory requirements, defeating the point of a lightweight computing model.

Maybe my fears are unfounded,  or will be addressed when 4G (WiMax or LTE) appears, but I still can’t see full coverage if you’re stuck up a mountain somewhere miles from the nearest transmitter.  Still, I await this project with interest and see how it pans out in the battle with Microsoft and the soon to be released Windows 7.   Its good to see Linux continuing to make inroads into the end user environment,  I just hope it doesn’t become the pervasive kernel standard to the detriment of other Linux projects.

BootNote:

You may have heard the tale attributed to various IT Thinkers of the early 20th century but generally now attributed to be Thomas Watson of the IBM Corporation.  The quote believed to be from 1943 goes “There is maybe a world market for maybe five computers”,  ironic considering how ubiquitous computing now is.  But the point is that Google is now building a single massive global computer (a mainframe if you will) that we all have access to via what could be pretty ‘dumb’ devices.  This computer currently runs the worlds most popular search engine,  it hosts videos,  allows you to go shopping and send Emails.  In 50 years time,  will computing be done not at the desk, but in ‘the world’?  Will we access the internet in the same manor that we access electricity today?  We currently plug an electrical appliance into a socket and it works using electricity.  In the future, will we plug a terminal into a socket and the internet ‘just works’.   Don’t be surprised if we move to a “pay for what you use” ‘net where you don’t buy access to the grid, but you do buy access to the resources you use just like the utilities of today.

Adventures in Time & Motion.

And now for my next experiment…

I found out a while ago (and have only just got round to trying it) that its possible to connect a Nintendo Wiimote to a computer using a Bluetooth Adapter.   The Wiimotes work via Bluetooth, so when you scan them they appear as HID’s and have a pin code of 0000.

So what's the point?  Well, primarily it makes it an excellent (and relatively inexpensive) remote control, ideal for a media centre or presentation controller.  Without using the motion sensors, the full set of buttons work natively on the controller to navigate, click forward & back, and exit (you can map button's to key's with software).

What is really cool is when you bring in the motion tracking and infra-red tracking elements.  The Wiimote (as I'm sure you're aware) has a 3 axis accelerometer and an infra-red tracking camera built in, which allows it to to know where it is in time and space.  Some clever developers, the most famous of whom is Johnny Lee, an MIT student who has written some excellent interface software to demonstrate the capabilities and possibilities of the hardware.

Some examples include using an IR-Led array and some reflective tape to provide an interface along the lines of "Minority Report",  building a cheap IR Pen to provide multi-touch interactive white boards, and my personal favourite, head-tracking to give 'virtual windows' into the world using this inexpensive technology.  You should checkout his website HERE where he discusses the capabilities and allows you to download the software.

Be sure to checkout the videos on YouTube - some impressive stuff indeed!

My Bargain of the Century…

I popped into my local Curry’s last night to look at headphones (see my previous thread on this topic) and whilst perusing the many isles where they store them, I happened to note the DAB radio section, and specifically the DAB/Wifi radio section, where upon I glanced upon the product you see on the right.

Its a Revo iblik wifi and its a clock radio that does…  FM, DAB, Internet Radio, Podcasts, UPnP media streaming, iPod dock and generic MP3 support.

Now, I’ve not got an iPod (having a deep mistrust of Apple products), but I have got access to all of the others.  And I have to say its amazing – rapid to setup, easy to use, excellent sound from something so small, it doesn’t look like a toaster or something from the 1950’s (why do DAB radio’s seem to have this requirement?) and works (so far) flawlessly.  I suppose that it shouldn’t be surprising how good it is when it retails at play.com for £169.99.  Which is why its a bargain when I picked up an ex demo unit for just £29.99. Unboxed, it was just the unit & the power wart, where as the boxed unit comes with…  A box,  remote control, stereo cable and manuals.  Well, the box goes straight in the bin anyway, the manual’s are available online, I’ve got stereo cables knocking around and the remote can be bought for the princely sum of £10 from the manufacturer’s website, should I feel the need to be able to control my alarm clock from the bed with a remote rather than having to reach my arm to tweak it.

And it seems like a good investment – the Government seem to be pushing to take the FM frequencies off the BBC and force them to use DAB only (more on the Digital Britain report later), but not only do I get local radio, but I can listen to radio you might not normally hear – last night I listened to RTL Germany,  the Sacramento fire department and the Atlantic radar control.  Amazing stuff, and recommended if you can find it at 1/5th the price it should be!

Oh yes, and I didn’t pickup any headphones. :)

NVidia Cards on Debian (specifically Lenny)

Anyone running Debian Xorg will know that getting accelerated Graphics drivers working (beyond using the nv driver) is more complicated than achieving world piece. I tried the official nvidia drivers, I tried the packages within the repositories, all to no effect.  However, http://desiato.tinyplanet.ca/~lsorense/debian/debian-nvidia-dri-howto.html has perfect instructions which worked first time.

Just a bit of detail around my card for reference:

• Nvidia GeForce 6200 128mb AGP.
• Debian Lenny (v 2.6.26-1-686)
• Intel Desktop Mobo (An 82865 IIRC).
• Gnome Desktop

I hope this is of use to someone, as I lost many an hour trying to get all other instructions working.  Also, for reference,  I gave up trying to get an ancient  Riva TNT2 working, but the 6200 is an excellent upgrade, and pretty cheap too.   It also means I'm able to decommission a PC from the living room, and hopefully I'll be able to get XBMC compiled and working on it too.

What time is it?

And the answer to the question is NOT Chico time.  The correct answer is 'it depends'.  One of the core network services, but one that often gets overlooked is having an accurate time across all IT systems.  Back in the time before the railways,  the time of an area was generally set by sunrise and sunset.  This was fine when people only travelled a few miles each day.  But when you could journey several hundred miles across the country in a day, you could end up with massive problems, especially when you wanted to catch a connecting train.  If your intermediary station was on a different time to your watch and the timetable set at your departing station, you could be in all sorts of trouble.  Hence the adoption of GMT and 'Railway Time',  a commonly agreed standard time across the country and around the world, regardless of your location. 

And so to the modern age of time synchronisation across a computer network.   Why is this so important you might ask?  Well, having the correct time is less important than making sure every computer clock is set the same.   However, it does make it easier if everything is set to the correct time as well.  From allowing people to login at certain times of the day, to analysing logs from different devices as you pass through the network, if everything is on the same time,  correlating activity can be an easier task if everything is synchronised.  Back before the days of flexible working where you were expected to clock on at 9am and clock off at 5.30pm, you needed to know that you were on the same time as the boss, otherwise there'd be hell to pay!

The company I work for operates two time servers.  These servers run network time services which check both with time servers out on the Internet and each other to decide on the correct time.  The reason for this multiple checking is that as time data is sent across the Internet, the inherent time delay and number of hops can skew the time out leading to inaccurate information.  However,  the accuracy shown is generally accurate to about 10ms,  plenty enough for what we require.  Users of more accurate time servers such as finance services and broadcasting have access to far more accurate time sync services and generally synchronise with what is known as stratum 0 devices. 

Time servers are referred to being in stratum groups - Stratum 0 are the 'core' clocks, be they atomic, gps or radio clocks and display the most accurate time available.  These are generally not available to the network, but are connected to Stratum 1 servers.   These level '1' devices receive the time feed from the 0 level, and then feed that down to Stratum 2 computers.  Our time devices are Stratum 2 devices synchronising with several stratum 1 servers.   These servers also check the time with each other to verify that neither are wildly off.  The companies servers, routers and other 'core' devices query these level 2 servers and become Stratum 3 devices.  Finally, your computers which synchronise with these servers become Stratum 4 devices.   It is possible to have upto 256 stratum levels, but generally 5-6 is seen.   However, stratum level DOES NOT INDICATE ACCURACY.  Because of the multiple check paths available to lower stratum devices, these can be more accurate than individual level 0 or 1 devices, because of this compare and contrast work that they do.  It simply notes how far 'away' from the central clock it is.

We could reduce the layers of stratum services within the company - the easiest way to do this is to invest in GPS time clocks.  GPS navigation requires each satellite to know precisely what time it is, in order that the receiver can compare the signals and work out how far away each satellite is, and therefore where it is in the world.  This can be harnessed using high quality GPS receivers to calculate the time, and passed to local stratum 1 servers, for example in the core data centres.  But is it worth it?  Probably not.

Another key requirement of the network time service is to ensure the clocks are regularly synchronised.  Computer 'Real-time-clocks' can be wildly inaccurate, often losing or gaining several seconds per day, especially when machines get switched off and on.  Our Stratum 2 servers synchronise every hour with the Internet and themselves, but generally lower stratums update every 4, 8 or 24hrs, or when machines are first switched on.   If you find that your clock is inaccurate, and you do not have the ability to update it,  try rebooting your computer and it should come back into line.

Network time is stored as UTC (Coordinated Universal Time) which to the casual user generally means GMT. This means that no matter what time zone you are in, or if its daylight savings,  your computer will be able to work out the correct time for your location, even if you're based in Australia, the Falklands Islands or Central London (provided the time zone database on your computer is accurate).

So when you put your clocks forward this weekend (1am Sunday becomes 2am Sunday), at least you know that the clock at work SHOULD be correct.

Asterisk*

This week seems to be a week of me telling you about my new toys.  Today's new toy is Asterisk, a Linux software PBX designed to offer the features of a large expensive phone switch for the grand sum of free.

It offers many features found in proprietary PBX systems such as Voice mail,  IVR menus, call distribution and routing, and conference calling.  You might ask why I might require such a device, and its quite simple really.  My PSTN phones are on their last legs, however, for the price of a replacement phone,  I have been able to buy a PSTN interface card which allows me to receive and send phone calls into the Asterisk system.  Asterisk is then able to convert this 'old tech' call into a digital format and send it over an IP network, which is something that I've invested in heavily at home, having excellent wireless and fixed distribution around the house.  Not only that, but because a call becomes 'VoIP',  it means that I can in theory route phone calls out over the Internet to my 3G mobile whilst I'm out and about.  Because of this pervasive network, it means I can configure each computer, smart mobile with a client to act as a phone, or buy an IP phone for not much more than a standard phone, but with all the extra functionality available.

I am also able to take advantage of least cost routing, meaning that calls that qualify for 'free' in BT's Option 2 profile can go out over the PSTN, whilst International or 'peak hour's' calls can be routed via a more inexpensive or free provider using Voice over IP services, for example Skype.

One nice features that I'm looking forward to is being able to filter out calls, so callers that withhold their numbers will get played a message advising that I am a member of the TPS and if their call is in relation to sales, to remove my number from the list.   If they redial and present an unrecognised number, they will be sent to a mailbox to leave a message where I can decide whether I want to talk to them or not.  And finally, callers with recognised numbers (friends and family) will be put thru to the 'logged in' phones with their name and caller ID displayed to them.  Plus, because my satellite box is network connected, I can popup a caller ID on screen. If an answer isn't forthcoming within say 30 seconds,  they will be put to a mailbox for later call back. 

Any calls sent to a voicemail box, because they're recorded digitally, can be forwarded via e-mail, to be picked up whilst I'm out and about.  And because of the ability to forward calls across the Internet, my family can have one number and know that they should always try and contact me on that, no matter where I am. 

Having only received the PBX card yesterday, I've only got the inbound/outbound dialling and mailbox handoff working, but all of the above features are possible with a little configuration, so hopefully a few evenings configuration and testing should get me pretty much there!   Huzzah!

WeekEPG Database Structure

So, I've spent some of the weekend figuring out the WeekEPG data structure, based on the files on my dreambox.  Here is a note of my discoveries so far.

Main 'Config' Files.

On my dreambox, the main config files are stored in /etc/ and we have:-

• channels.uk/it - Where the channel listings are stored.  Data format seems to be 
  1;2002;BBC 1 London;101;189d:7fd:2
  Scan Enabled (1=Y); Unknown (but suspect its just a unique id);Channel Name; Assigned Channel Number; ServiceID(in Hex):Transponder:NetworkID (2=Sky).
• Dictionary.uk - Appears to be a translation table from the binary stream into text.
• Themes.uk/it - Theme definition (Kids/Sports e.t.c.)
  THM;0;40;Children
  SUB;1;41;Cartoons
  Type (Main Theme or SUB theme);Whether Listing is a main or sub theme;Theme Unique ID;Theme Name.

Data Files

Stored in /hdd/weekepg (or where you've defined it in the setup).

• titlesx.gep - Stores the program names, titles e.t.c. Note, there can be numerous ones of these.
  1A660416<>1231680600<>201<> Inside the Bermuda Triangle<>3600<>60
  Unique Program ID <> Time of program start (Unix Time) <> Channel Number (from the channel.uk def) <> Program Name <> Length in Minutes <> Theme (from Themes.uk)
• summariesx.gep - Stores more detailed descriptions of the program, including summary description. Note, like titlesx, numerous ones of these.
  728056F<>A free holiday to Boston, New York or Barbados, among others, is yours when you book a featured stay and cruise holiday to Orlando and the Caribbean in this great Sky Travel sale offer. 
  Unique Program ID (matches titlex.gep UID) <> Detailed Description (may include subtite, series link and other information at the end of description.

So that's what I've discovered so far about it.  I'm currently writing a PHP loader to import this data into a MySQL database, so that I can then write a further PHP script to be able to publish XMLTV formatted files for interpretation by MV and MythTV.  WeekEPG rocks, but I prefer the interface with MV, and could really use the data within MythTV for the Freesat Channels.

Not at one with XEN.

So, I've spent the last few weeks setting up (READ: FIGHTING WITH) Xen-Source on a server we have here.

It took me a few days to get my distro of choice installed on the HP DL380, given that (as of writing) etch isn't a directly installable distribution. I eventually figured out that to get the damn thing running, you have to download the Deb 3.1 install cd hidden away on the HP site, install the server from that. At that point, it was an upgrade to Etch via aptitude, THEN an upgrade to Xen also via the repositories.

After doing all the necessary tweeking, it was time to build a DomU, so first off a Deb Etch 'test' machine. That took me a good couple of hours to note that because we were behind a proxy, each time we wanted to use the image scripts, I had to do the export_proxy=http://.... command.
After this all was well.

So now we get to the main gist of my post. Virtualised Windows. Ghaaa....
Starting off with XP for a test image, its taken me about 4hrs of looking around, perseverence and banging my head against the wall to get the damn thing booting. First of all the config. This took a while to even get the damn thing to boot without crashing (and using VNC to allow remote control). This is the config I ended up with:-

import os, re
arch = os.uname()[4]
if re.search('64', arch):
arch_libdir = 'lib64'
else:
arch_libdir = 'lib'

kernel = "/usr/lib/xen-3.0.3-1/boot/hvmloader"
device_model = '/usr/lib/xen-3.0.3-1/bin/qemu-dm'
builder = 'hvm'
memory = '512'


name = "XenXP1"
vif = [ 'type=ioemu, bridge=xenbr0' ]

disk = [ 'phy:/dev/fs1/Win2003_TST,ioemu:hda,w', 'file:/etc/xen/isos/win/xp/XP_SP2.ISO,hdc:cdrom,r',]

boot="dc"
sdl=0
vnc=1
vncdisplay=3
vncunused=1
vncconsole=1
vnclisten="0.0.0.0"
vncpasswd = "apassword"
acpi = 1
apic = 0
stdvga=0
serial='pty'

usb=1
usbdevice='tablet'

on_reboot = 'destroy'
vcpus=1
ne2000=1

Which would at least allow me to boot into the blue windows installer. However, once that had run through, it would just sit there with a black screen doing nothing else (although xm list still had a status of running). I eventually stumbled on this page http://lists.xensource.com/archives/html/xen-users/2007-02/msg00249.html which had the solution! Force Windows to install as a standard machine. This has now got me to the GUI with 34 minutes to run. Fingers x'd thats the solution.