23 April 2017

23rd of April

Farm Status
Intel GPUs
All running Seti overnight

Nvidia GPUs
Two running Seti overnight. Other two are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
The WD Red hard disks turned up. I expect the Drobo that they are destined for will arrive late next week.

I also sold off a couple more GPU's and the last remaining motherboard.

Now that I have sold off  most of the excess hardware I will look at replacing the i7-5820K machines (6 core/12 thread) with AMD Ryzen's. I'm looking at the Ryzen 1700 (3Ghz) but also only 65 watts. The motherboard would be an ASUS Prime X370 with 32GB of DDR4 memory. I will keep the GTX1060's that are in there as they are the most up to date GPU's that I have.


Transition to Linux
In the blog post prior to this one I covered the steps used to converted two of my Nvidia GPU machines to Linux. They can run CUDA, OpenCL and CPU work now. I have two more to convert and that is all the farm converted to Linux apart from two windows-based machines. One is the file server and the other is a laptop that I use to monitor the farm using BOINCtasks.

16 April 2017

Setting up a Linux CUDA cruncher

Rather than using an outdated Ubuntu which seems to be the current suggestions, this is what I used with Debian. This will get you the latest Debian (Jessie), latest kernel (4.9) and the 7.6.33 BOINC client. You should be up and running with a CUDA and OpenCL capable machine after doing this.


Part 1 - Install Debian
I used the Debian 8.7 net install for this. You’ll need a thumb drive or a blank CD. Download Debian from http://www.debian.org/distrib/ and write the ISO image to CD or thumb drive.

Boot off the thumb drive or CD. It will start up the Debian installer

Install Debian. It will ask a bunch of questions like language, country, etc and prompt for partitioning your disks. When it asks for software to install select SSH server and whatever desktop you prefer and remove all other selections. Once done it will prompt for a reboot.


Part 2 - Install Nvidia software
Login as root, open a xterm window and type the following commands:

cd /etc/apt

nano sources.list (nano is a text editor)

Change “jessie main” lines to “jessie main contrib non-free” and add a jessie-backports line. It should look like this when you're done. I'm using httpredir as it will pick the fastest server.

deb http://httpredir.debian.org/debian/ jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-backports main contrib non-free

Exit out of nano and save the file (Control-O followed by Control-X)

apt update

apt install –t jessie-backports firmware-realtek (if needed). Most of my motherboards have Realtek chips.

apt install –t jessie-backports linux-image-amd64

apt install –t jessie-backports nvidia-kernel-dkms nvidia-smi nvidia-xconfig

apt install –t jessie-backports nvidia-opencl-icd (if you want OpenCL support)

nvidia-xconfig

  
Part 3 – Install BOINC
apt install –t jessie-backports boinc-nvidia-cuda boinc-manager

sync

reboot

14 April 2017

Good Friday 2017

Farm status
Intel GPUs
All running Seti work

Nvidia GPUs
Did a couple of GPUgrid work units earlier in the week. Currently they are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
I have ordered three 4TB WD Red hard disks for the Drobo 5N2. The Drobo isn't due here until the end of the month. The price per terabyte increases with the larger capacity drives. Two 4TB drives are still cheaper than one 8TB drive. Seagate have a 12TB drive but they cost over a thousand dollars.

A couple of the Pi's were playing up so I had to re-image them. Unfortunately when attaching them to the Einstein project they decided the program executable was corrupted and then trashed their work units. A project reset in BOINC fixed the corrupt program. It did this on both of them so I think the checksum value given for the executable is probably wrong rather than a download issue.

I've rescheduled the ADSL connection for the 26th of April so hopefully it will get installed this time. The last technician didn't have a key to the communications cupboard.


Linux transition
The Intel GPU's are doing work faster than when they were windows machines. All 8 of them are churning through Seti work. I am trying to get the credits similar for the 3 projects they've been set up to run (Asteroids, Einstein and Seti).

One of things to try on the transition to Linux is getting the Nvidia GPU's working. Debian have Nvidia drivers in their repository so hopefully they will work. Most of the guides I have seen install older versions of Ubuntu (14.04) or Mint which is based upon Ubuntu.

I also want to look at running CPDN jobs however their programs are all 32 bit so that usually causes problems when running on a 64 bit Linux platform.

08 April 2017

8th of April

Farm status
Intel GPUs
3 running Seti work. The rest are off.

Nvidia GPUs
All off

Raspberry Pis
All running Einstein BRP4 work


Other news
I sold off one of the ASUS P6T's. The other one and a GTX670 are on eBay ending tomorrow. I also sold off some Seasonic power supplies. I have interest in the remaining motherboards and a couple of GTX750Ti cards. That still leaves a bunch of other parts and the two i7-5820K system to sell off.

I had a phone company technician come out to install a second ADSL line. Unfortunately he didn't have a key to the communications cupboard. Its meant to be a standard set for phone companies but it seems they don't come prepared.

I'll put up a few more graphics cards on eBay next week to get rid of them.

I've ordered a new Drobo 5N2 to replace my Drobo 5N with. I need to order some larger WD Red drives to go in it so it can backup the storage server. Once its up and running I will sell off the 5N.

02 April 2017

2nd of April

Farm Status
Intel GPUs
All except one currently running Seti work. They were running Asteroids work before.

Nvidia GPUs
All off.

Raspberry Pis
All running Einstein BRP4 work


Farm news
I've changed all except one of the Intel GPUs over to Linux. That's 7 of them. The last one had an SSD + HDD which I had to swap out with the proxy servers SSHD. Its now the same configuration as all the other Intel GPU machines and currently installing Linux. I had to rebuild the proxy server.

I found that installing Beignet (Intels OpenCL drivers for Linux) doesn't work if you are on the 3.16 kernel. It will install but clinfo is unable to detect the device. Updating the kernel to a 4.x one seems to have resolved that. I can now use the iGPU but it slows down the CPU so is not normally worth the effort.

For the moment I am leaving the Nvidia GPU machines running Windows but plan on selling off the two GTX970's and also the two 6 core/12 thread machines which currently have GTX1060's. I'm going to keep the GTX1060's but will need to put something else in there in order to sell them. I will need to investigate running the GPUs under Linux and what driver combinations work.

The computer shop rang regarding the storage server. They have received it from their supplier but it seems to only have SAS expanders for each row of drive bays. Its got 12 drive bays but each set of 3 has its own cabling. The hard disk controller has only a single SAS connector meaning it can only run 3 drives. I'll have to chase them up next week. It seems the shop don't have experience with storage servers.

19 March 2017

19th of March

Farm Status
Intel GPUs
All off at the moment. They have been running overnight.

Nvidia GPUs
All off at the moment. The two GTX1060's were run overnight.

Raspberry Pis
All running Einstein BRP4 work


Linux testing
I did some more testing with Linux giving various VM's a workout. I have a Debian Jessie VM, a Debian Stretch VM and a Mint VM. I'm in a position to switch the Intel GPUs over to Linux without too much drama. About the only problem I see is each machine will get a new identity and lose its history of work done.

It seems Microsoft has started giving messages about unsupported CPU's on earlier versions of Windows with the latest CPU's which includes the AMD Ryzen. That means all new equipment I buy will probably be running Linux. Not the smartest idea from Redmond. It will be interesting to see how much market share they lose.


Storage Server
It is still on its way. I have already picked up a bunch of new 4TB hard disk drives to go in it. I have also ordered a 10Gbit network card to put in it. That means I'll have to get a couple of switches that support 10Gbit before I can make use of it. I am thinking of a 1Gbit switch with a 10Gbit uplink for most of the crunchers and a 10Gbit main switch. All my machines (apart from the Raspberry Pis and the AlphaServer) have 1Gbit network ports, some have two.

I need to make some room for the new server. I'm expecting to get rid of a number of CM Storm Sniper cases. Two of them still have ASUS P6T's and Corsair H80 water cooling in them, the other parts have been cannibalised. The other two are empty.

After that I need to sell off the existing i7-5820K's (I have two) before I can order some AMD Ryzen's to replace them with. Then there are all the old graphics cards that I need to get rid of...

07 March 2017

7th of March

Farm Status
Intel GPUs
All except one running Seti work

Nvidia GPUs
Ran all of last weekend. Currently off.

Raspberry Pis
All except two running Einsten BRP4 work


Linux testing
In my hunt to find alternatives to Windows I have been installing various Linux flavours on a couple of machines. My current leaning is towards Debian. I'd like to get an up to date kernel and they have an up-coming release called Stretch. When exactly it will be available is a guess, but maybe April 2017. Why do I need the later kernel, well those AMD Ryzen machines need an up to date kernel to correctly run tasks on them.

I spent a lot of time last weekend and again this weekend installing and re-installing to find out that either upgrading to Stretch or installing via the Stretch net-install (RC2) doesn't work and you lose the desktop.

At the moment I've setup a virtual machine and I'm on the 2nd reinstall of Debian after it decided to remove my entire desktop due to doing apt-get remove gstreamer* and apt-get autoremove commands.

I did have a play with Mint 18.1 (Serena) which is visually great however I don't want a full-blown desktop system just to run on the number crunchers and possibly storage servers. A light-weight desktop is enough for my needs. I can even work with no desktop and use the command-line if needed.


Storage server
I have ordered a bunch of 4TB hard disks and a 2U 12 bay storage server. Its taken almost 2 weeks of back and forth with the shop to get it configured the way I want.


I currently have a Windows-based file server with a RAID controller and 4 x 4TB hard disks and while it works fine expanding its capacity is rather difficult. As it turns out the one I am buying uses the same motherboard as my existing file server, so I may just rebuild it later into the same configuration and use it to backup the new one.


Ryzen woes
After the PC Case Gear payment portal decided it didn't like me being behind a corporate firewall I haven't placed my order. That turned out to be a good thing because there are a couple of major issues with them.

First the motherboard people are having to correct issues with their BIOS so most motherboards are hard to find at the moment.

Second the windows task scheduler doesn't understand they are hyper-threaded chips and so it moves tasks around more than it should effecting performance. That will probably require Microsoft to provide an update. How long and which versions of windows it will be available for is unknown. Linux (if you have an up to date kernel) doesn't have this issue.

Once these issues are ironed out I think they'll provided a great replacement for my 6 core/12 thread machines and I will place my order. I need to sell off my old hardware to make room for the new stuff.

26 February 2017

26th of February

Farm Status
Intel GPUs
All running Einstein Multi-directed Gravity Wave work

Nvida GPUs
Two running Seti work

Raspberry Pis
All except two running Einstein BRP4 work. The other two (a Pi2 and a Pi3) are running Seti Beta work


Seti multi-beam apps for Rpi
There has been some progress on the Raspberry Pi app which is why I have a Pi2 and Pi3 running Seti Beta. We're currently testing an updated app that is slightly faster than the previous one as well as being able to work on the ARM v6 (Pi Zero, B and B+). If you're going to run on a ARM v6 apart from it being very slow it also needs a patched kernel. Even the ARM v7 is slow when it gets a VLAR (Very Low Angle Range) work unit but at least it can do them now.


AMD Ryzen
I've asked my usual computer shop to quote on a couple of Ryzen S7 1700 machines to replace the two i7-5820K gas-guzzlers. The parts showed up on the PC Case Gear website yesterday (due for delivery 3/03/2017). I'm looking at two for the moment but given they use a bit less than half of the wattage of my current i7's I could get more AMD machines. I would expect they'll be slower than the i7 but then I get two extra threads so it will probably produce around the same throughput.


Storage Server
I am also looking at updating the file server, or at least its storage capacity. Currently I have an Intel RAID controller in it that provides 4 ports and there are 4 x 4TB drives in there.

SuperMicro have a storage server which is a similar motherboard to what I already have with a built-in SAS controller and another 10 SATA ports on the motherboard provided by an Intel chipset and it comes with 12 (3.5 inch) hot-plug drive bays on the front and another two (2.5 inch) on the back. The bad news is they is they only support Windows Server (2008 or 2012) or RHEL on it.

There are a number of other options such as a SAS expander, another RAID controller, bigger drives and so on that I could use which I will look into.


ZFS on Linux
The file server had me looking at ZFS on Linux. My first attempt was so try and install it on a Pi2 which failed miserably. It downloads the code and you have to compile it. The compile failed for a couple of reasons:
  • Its written with AMD64 architecture in mind
  • You have to get the Linux kernel headers to compile it
The Pi2 and Pi3 are currently ARM (32 bit) architecture. Technically the Pi3 could run ARM64 architecture as it an ARM v8 CPU but the Raspberry Pi foundation don't have it available.

As for the Linux kernel headers one has to get the version from the Raspberry Pi foundation that matches whatever version they are on (mine currently have 4.4.38). That isn't a simple apt-get command, you have to back-track the 4.4.38 version to the git check-in to find it.

My next attempt was to install Debian Stretch onto one of the i7-6700's which sort of worked. It gave some errors at bootup. I guess it doesn't have the necessary drivers for an H170-Pro motherboard. As well as that the beignet-opencl-icd didn't support the HD Graphics 530 that they have so I gave up on that one and reinstalled Win7 on it.

Ubuntu has a pre-compiled one in their repository so you don't have to compile your own, however there is some debate around the licensing of it.

28 January 2017

28th of January

Its still hot here in Sydney. We've had a few cool days where I have managed to run some Einstein or Seti work. The Einstein CPU tasks take about 16 hours so I am tending to run Seti as they're somewhere between 1 and 3 hours depending on the work unit.

I purchased another 5 port USB charger which I am using to power four Raspberry Pi2's. Its an Astrotek brand with a part number of AT-UPS-008B. Provided you only use 4 out of the 5 ports its fine. I have another one running four of the Pi3's. It might actually be able to run five (it reduces power output to each port when you plug in a 5th device) but I haven't tried five.

17 January 2017

17th of January

Its summer here in Sydney and its hot. Today was 32 degrees (C). The only things running on the farm are the Raspberry Pi's.

I got another 5 port USB charger. The ones I'm using to run the Pis are an Astrotek brand. Its a 5 port 8 amp unit. If you only plug 4 devices into it it can provide 2 amps per port. If you plug a 5th one in it drops the available power so I normally run four. The second charger is running the four Pi2's that have been sitting around gathering dust. I get a RAC (recent average credit) on Einstein BRP4 of about 300 from a Pi2's and 515 from a Pi3.

There are some rumours of a Pi4. There is nothing official at the moment but historically the Raspberry Pi foundation announce new models in February. The suggestion is it will have an ARM Cortex A53 processor at 1.8Ghz and 2Gb of memory. There are contrary suggestions that there won't be a new one for a few years as the foundation ordered 600,000 Pi3's and would need to sell these first.