17 June 2017

17th of June

Farm status
Intel GPUs
All running Seti work

Nvidia GPUs
All running Einstein gravity wave work

Raspberry Pis
All running Einstein BRP4 work


Rpi news
I got 3 more Pi3's. One replaced the faulty #9. I added fans to the tops of a couple of the official Pi cases. Unfortunately I didn't get the hole centered on one of them so that is one case messed up. A second Pi3 (with fan) has been installed and is now crunching away. I will add the other new one into the mix which will bring the bramble up to 10 Pi3's crunching.

The Pi2's have been retired. The bramble is close to 2 million credits of Einstein crunching.


Other news
Debian Stretch is being released as I write this. I will wait a week or two before looking at upgrading crunchers. Jessie is still supported for another year. It may be better to clean install Stretch rather than upgrading them from Jessie.

I tried the GPS receiver on a couple of machines, one Intel and a Raspberry Pi, it would happily talk to the computer but it seems it couldn't get a fix on the satellites. I had it stuck to the inside of a skylight window. Its not weatherproof so I can't just hang it outside. I will try it outside for a short while to see if I can get it to work.

10 June 2017

10th of June

Farm status
Intel GPUs
Running Einstein gravity wave and Seti work

Nvidia GPUs
The Ryzens are running Seti work. The i3's are running Einstein gravity wave work

Raspberry Pis
All running Einstein BRP4 work


Rpi news
It seems there is a Trojan out that specifically targets the Raspberry Pi. Apparently a lot of people don't change the default password. Here is a link to the article: http://www.guru3d.com/news-story/new-linux-trojans-installs-crypto-currency-mining-software-on-raspberry-pi.html

The copper heatsinks from Enzotech arrived yesterday. I have ordered 3 more Pi3's and a couple of cases.


Electricity price increase
My current electricity retailer has announced a 16% price increase for electricity and 9% for gas from the 1st of July. There are a couple of other retailers I could switch to. I will compare them but I expect they will all increase around the same time.

The electricity price increase would mean the Ryzen upgrade (from a 1700 to 1700X) would not be cost effective. It uses another 30 watts for around a 10-12% increase in output. I might have made the right decision about which Ryzen chip to get after all :-)

I have seen a natural gas powered generator around the size of a washing machine for business use but they have a high purchase price. A diesel or petrol powered generator is going to be cheap to buy but cost a lot to run. Its time to look into a solar and battery solution.


Other farm news
My current planning has me looking at another pair of Ryzen 1700's. I have spare cases, hard disk drives and graphics cards for them to keep the cost down. This would free up the i3's for something else although I don't currently have anything in mind for them.

04 June 2017

4th of June

Farm Status
Intel GPUs
All running Seti work

Nvidia GPUs
All running Seti work (after running Einstein gravity waves during the week)

Raspberry Pis
All running Einstein BRP4 wotk


Other news
Last weekend the farm was off, except the Raspberry Pis, due to having to attend an out of town wedding. The Ryzens and the storage server were delivered the day prior to the wedding. Unfortunately I didn't have time to configure them before leaving.

The first Ryzen was setup last week and spent the week running different projects before switching to Seti work. The second machine was setup yesterday and is also doing a range of projects. Performance is a bit slower than the Intel i7-5820K's that they replaced. I am considering replacing the Ryzen 1700 with 1700X CPUs (3.4GHz base clock) but have to weigh up the extra power consumption (another 30 watts). I will also need to replace the heatsink but can reuse the AM4 mounting kit. Each Ryzen also has a GTX1060 installed.

The firmware still hasn't caught up with the Ryzens and Linux throws a number of errors when starting up even though I have a 4.9 kernel installed and the latest firmware from ASUS. I expect they will need a few updates before most of the bugs are fixed.

I've got another batch of 14mm copper heatsinks on the way from Enzotech, so I can look at replacing the two Raspberry Pis mentioned in my last blog post. I still need to look at a better case for multiple Pis.

20 May 2017

20th of May

Farm status
Intel GPUs
Spent all of last week running Einstein gravity wave work. Now back to Seti

Nvidia GPUs
Running Einstein gravity wave work (CPU only)

Raspberry Pis
Running Einstein BRP4 work


Other news
Now I have the second internet connection I split the network into two. The Raspberry Pis are now over on the 2nd net. It took a while to unhook the Pis from the main network with network cabling and having to dig out an unused switch. Software-wise I spent a bit of time removing dependencies on machines that are now on the main network and therefore inaccessible. Two of the Pis have Pi Drives (USB hard disks) so one of them was turned into a proxy and time server.

One problem I have with the Pi's is they don't seem to broadcast their device name so I can only reference them by their IP address, which means they have to be assigned a static IP address. Sure they can get a dynamic IP address from the router but I can't access them by name, only by their IP address. The PC's running Linux don't have this problem so it seems something unique to Raspbian.

I will order a couple more Pi3's. I need to replace a faulty Pi3 and replace the one that got used for the proxy/time server. I also need to order more copper heatsinks from Enzotech.

One of the other projects I am doing is a more accurate time server for the main network. I have purchased a USB GPS receiver which can be used to obtain accurate time. It needs a view of the sky and the machine I am intending to connect it to is furthest away from the window/skylight. Today I have ordered a USB extention cable so I can connect it.

The pair of GTX1060's that I ordered for the Ryzen machines arrived during the week. Unfortunately the Ryzen's still haven't been built due to delays with getting the heatsinks. Maybe next week.

13 May 2017

13th of May

Farm status
Intel GPUs
All doing Einstein Gravity wave

Nvidia GPUs
Doing Einstein Gravity wave (CPU) plus Seti GPU work

Raspberry Pis
Doing Einstein BRP4 work


Project news Einstein
They had a problem with their first tuning run so we've got updated apps and are doing a second tuning run.


Other news
Still waiting on the Ryzen builds. The computer shop were waiting on Noctua heatsinks to come in. It would have been nice to see how they perform on the Gravity wave tuning run as its CPU only. Hopefully they'll be ready next week.

The new Drobo arrived. I had a bit of a problem with it due to taking the mSata SSD from the old Drobo. A software reset fixed that. I think its 50% faster than the old one. I need to list the old one on eBay now that I've cleaned off the hard disks.

The second ADSL connection is running however I need to work out how to load balance the two internet connections. I know TP Link make a $130 load balancer device which might be how I'll run it. I need to switch the main internet connection back to a standard ADSL first. It seems the wiring between the communications cupboard and the house isn't too good because they can get 18Mbit to the comms cupboard, but to the house only gets 13Mbit.

07 May 2017

7th of May

Farm status
Intel GPUs
Mostly Seti with a batch of Einstein work

Nvidia GPUs
Mostly Seti with a batch of Einstein work

Raspberry Pis
All running Einstein BRP4 work


Project news Einstein
They've started another run of their gravity wave work units. Currently its a tuning run to get an idea how long each type of CPU takes and iron out any bugs with the apps. There are two types of work unit called Hi and Lo. When I ran all 8 cores on the i7-6700's the Hi ones were taking 12-13.5 hours. When running half the cores they were taking 9 hours.

My i7-6700's only have 16Gb in total. The Ryzen's will have 32Gb. The higher frequency work units need 2Gb of memory each so I can only run 7 on the i7's and 14 on the Ryzen's.

The Lo work units are sent to slower machines and don't take as long. The i3-6100T's were talking around 3.5 hours when running all cores. These work units also use much less memory (around 270Mb).


Other news
The two AMD Ryzen 7 motherboards, CPUs and memory kits arrived and I dropped them off at my usual computer shop. I supplied the GPUs and hard disks and they are supplying the case, power supply, cooler and assembling it. Hopefully they will be ready next week.

The two i7-5820K's were sold off and have gone. That leaves the farm with two working GPU's at the moment. I kept the GTX970's which will go into the Ryzen's to start with and get swapped for the GTX1060's when they arrive, supposedly on the 18th.

The Drobo 5N2 is due to arrive next week some time.

30 April 2017

30th of April

Farm Status
Intel GPUs
All running Seti work

Nvidia GPUs
Running Seti work

Raspberry Pis
Running Einstein BRP4 work


Transition to Linux
The last two machines were converted over to Linux without any issues. I simply follow the process I documented before. That means there are no Windows crunchers in the farm now.


Other news
The same person that purchased all the motherboards also wants to buy the i7-5820K machines. I decommissioned them and removed the GTX1060's on Friday. I will be replacing them with a pair of AMD Ryzen 1700 machines. The GTX1060's will go into the Ryzen's when I get them.

I have also ordered another pair of GTX1060's. These will initially be used to replace the GTX970's. Longer term I will probably replace the i3's with another pair of Ryzen machines.

My original idea was to have a low powered CPU (the i3-6100T is only 35 watts) and a pair of high performance GPU's. Unfortunately the i3 doesn't have enough PCIe lanes  to run two GPU's at x16 speed. It seems to provide one slot at x16 and the second at x4 and this is on a Z170 chipset motherboard. The i3 does have built-in graphics and the motherboard is mATX sized so I may be able to use them for some other purpose.

23 April 2017

23rd of April

Farm Status
Intel GPUs
All running Seti overnight

Nvidia GPUs
Two running Seti overnight. Other two are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
The WD Red hard disks turned up. I expect the Drobo that they are destined for will arrive late next week.

I also sold off a couple more GPU's and the last remaining motherboard.

Now that I have sold off  most of the excess hardware I will look at replacing the i7-5820K machines (6 core/12 thread) with AMD Ryzen's. I'm looking at the Ryzen 1700 (3Ghz) but also only 65 watts. The motherboard would be an ASUS Prime X370 with 32GB of DDR4 memory. I will keep the GTX1060's that are in there as they are the most up to date GPU's that I have.


Transition to Linux
In the blog post prior to this one I covered the steps used to converted two of my Nvidia GPU machines to Linux. They can run CUDA, OpenCL and CPU work now. I have two more to convert and that is all the farm converted to Linux apart from two windows-based machines. One is the file server and the other is a laptop that I use to monitor the farm using BOINCtasks.

16 April 2017

Setting up a Linux CUDA cruncher

Rather than using an outdated Ubuntu which seems to be the current suggestions, this is what I used with Debian. This will get you the latest Debian (Jessie), latest kernel (4.9) and the 7.6.33 BOINC client. You should be up and running with a CUDA and OpenCL capable machine after doing this.


Part 1 - Install Debian
I used the Debian 8.7 net install for this. You’ll need a thumb drive or a blank CD. Download Debian from http://www.debian.org/distrib/ and write the ISO image to CD or thumb drive.

Boot off the thumb drive or CD. It will start up the Debian installer

Install Debian. It will ask a bunch of questions like language, country, etc and prompt for partitioning your disks. When it asks for software to install select SSH server and whatever desktop you prefer and remove all other selections. Once done it will prompt for a reboot.


Part 2 - Install Nvidia software
Login as root, open a xterm window and type the following commands:

cd /etc/apt

nano sources.list (nano is a text editor)

Change “jessie main” lines to “jessie main contrib non-free” and add a jessie-backports line. It should look like this when you're done. I'm using httpredir as it will pick the fastest server.

deb http://httpredir.debian.org/debian/ jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-backports main contrib non-free

Exit out of nano and save the file (Control-O followed by Control-X)

apt update

apt install –t jessie-backports firmware-realtek (if needed). Most of my motherboards have Realtek chips.

apt install –t jessie-backports linux-image-amd64

apt install –t jessie-backports nvidia-kernel-dkms nvidia-smi nvidia-xconfig

apt install –t jessie-backports nvidia-opencl-icd (if you want OpenCL support)

nvidia-xconfig

  
Part 3 – Install BOINC
apt install –t jessie-backports boinc-nvidia-cuda boinc-manager

sync

reboot

14 April 2017

Good Friday 2017

Farm status
Intel GPUs
All running Seti work

Nvidia GPUs
Did a couple of GPUgrid work units earlier in the week. Currently they are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
I have ordered three 4TB WD Red hard disks for the Drobo 5N2. The Drobo isn't due here until the end of the month. The price per terabyte increases with the larger capacity drives. Two 4TB drives are still cheaper than one 8TB drive. Seagate have a 12TB drive but they cost over a thousand dollars.

A couple of the Pi's were playing up so I had to re-image them. Unfortunately when attaching them to the Einstein project they decided the program executable was corrupted and then trashed their work units. A project reset in BOINC fixed the corrupt program. It did this on both of them so I think the checksum value given for the executable is probably wrong rather than a download issue.

I've rescheduled the ADSL connection for the 26th of April so hopefully it will get installed this time. The last technician didn't have a key to the communications cupboard.


Linux transition
The Intel GPU's are doing work faster than when they were windows machines. All 8 of them are churning through Seti work. I am trying to get the credits similar for the 3 projects they've been set up to run (Asteroids, Einstein and Seti).

One of things to try on the transition to Linux is getting the Nvidia GPU's working. Debian have Nvidia drivers in their repository so hopefully they will work. Most of the guides I have seen install older versions of Ubuntu (14.04) or Mint which is based upon Ubuntu.

I also want to look at running CPDN jobs however their programs are all 32 bit so that usually causes problems when running on a 64 bit Linux platform.

08 April 2017

8th of April

Farm status
Intel GPUs
3 running Seti work. The rest are off.

Nvidia GPUs
All off

Raspberry Pis
All running Einstein BRP4 work


Other news
I sold off one of the ASUS P6T's. The other one and a GTX670 are on eBay ending tomorrow. I also sold off some Seasonic power supplies. I have interest in the remaining motherboards and a couple of GTX750Ti cards. That still leaves a bunch of other parts and the two i7-5820K system to sell off.

I had a phone company technician come out to install a second ADSL line. Unfortunately he didn't have a key to the communications cupboard. Its meant to be a standard set for phone companies but it seems they don't come prepared.

I'll put up a few more graphics cards on eBay next week to get rid of them.

I've ordered a new Drobo 5N2 to replace my Drobo 5N with. I need to order some larger WD Red drives to go in it so it can backup the storage server. Once its up and running I will sell off the 5N.

02 April 2017

2nd of April

Farm Status
Intel GPUs
All except one currently running Seti work. They were running Asteroids work before.

Nvidia GPUs
All off.

Raspberry Pis
All running Einstein BRP4 work


Farm news
I've changed all except one of the Intel GPUs over to Linux. That's 7 of them. The last one had an SSD + HDD which I had to swap out with the proxy servers SSHD. Its now the same configuration as all the other Intel GPU machines and currently installing Linux. I had to rebuild the proxy server.

I found that installing Beignet (Intels OpenCL drivers for Linux) doesn't work if you are on the 3.16 kernel. It will install but clinfo is unable to detect the device. Updating the kernel to a 4.x one seems to have resolved that. I can now use the iGPU but it slows down the CPU so is not normally worth the effort.

For the moment I am leaving the Nvidia GPU machines running Windows but plan on selling off the two GTX970's and also the two 6 core/12 thread machines which currently have GTX1060's. I'm going to keep the GTX1060's but will need to put something else in there in order to sell them. I will need to investigate running the GPUs under Linux and what driver combinations work.

The computer shop rang regarding the storage server. They have received it from their supplier but it seems to only have SAS expanders for each row of drive bays. Its got 12 drive bays but each set of 3 has its own cabling. The hard disk controller has only a single SAS connector meaning it can only run 3 drives. I'll have to chase them up next week. It seems the shop don't have experience with storage servers.

19 March 2017

19th of March

Farm Status
Intel GPUs
All off at the moment. They have been running overnight.

Nvidia GPUs
All off at the moment. The two GTX1060's were run overnight.

Raspberry Pis
All running Einstein BRP4 work


Linux testing
I did some more testing with Linux giving various VM's a workout. I have a Debian Jessie VM, a Debian Stretch VM and a Mint VM. I'm in a position to switch the Intel GPUs over to Linux without too much drama. About the only problem I see is each machine will get a new identity and lose its history of work done.

It seems Microsoft has started giving messages about unsupported CPU's on earlier versions of Windows with the latest CPU's which includes the AMD Ryzen. That means all new equipment I buy will probably be running Linux. Not the smartest idea from Redmond. It will be interesting to see how much market share they lose.


Storage Server
It is still on its way. I have already picked up a bunch of new 4TB hard disk drives to go in it. I have also ordered a 10Gbit network card to put in it. That means I'll have to get a couple of switches that support 10Gbit before I can make use of it. I am thinking of a 1Gbit switch with a 10Gbit uplink for most of the crunchers and a 10Gbit main switch. All my machines (apart from the Raspberry Pis and the AlphaServer) have 1Gbit network ports, some have two.

I need to make some room for the new server. I'm expecting to get rid of a number of CM Storm Sniper cases. Two of them still have ASUS P6T's and Corsair H80 water cooling in them, the other parts have been cannibalised. The other two are empty.

After that I need to sell off the existing i7-5820K's (I have two) before I can order some AMD Ryzen's to replace them with. Then there are all the old graphics cards that I need to get rid of...

07 March 2017

7th of March

Farm Status
Intel GPUs
All except one running Seti work

Nvidia GPUs
Ran all of last weekend. Currently off.

Raspberry Pis
All except two running Einsten BRP4 work


Linux testing
In my hunt to find alternatives to Windows I have been installing various Linux flavours on a couple of machines. My current leaning is towards Debian. I'd like to get an up to date kernel and they have an up-coming release called Stretch. When exactly it will be available is a guess, but maybe April 2017. Why do I need the later kernel, well those AMD Ryzen machines need an up to date kernel to correctly run tasks on them.

I spent a lot of time last weekend and again this weekend installing and re-installing to find out that either upgrading to Stretch or installing via the Stretch net-install (RC2) doesn't work and you lose the desktop.

At the moment I've setup a virtual machine and I'm on the 2nd reinstall of Debian after it decided to remove my entire desktop due to doing apt-get remove gstreamer* and apt-get autoremove commands.

I did have a play with Mint 18.1 (Serena) which is visually great however I don't want a full-blown desktop system just to run on the number crunchers and possibly storage servers. A light-weight desktop is enough for my needs. I can even work with no desktop and use the command-line if needed.


Storage server
I have ordered a bunch of 4TB hard disks and a 2U 12 bay storage server. Its taken almost 2 weeks of back and forth with the shop to get it configured the way I want.


I currently have a Windows-based file server with a RAID controller and 4 x 4TB hard disks and while it works fine expanding its capacity is rather difficult. As it turns out the one I am buying uses the same motherboard as my existing file server, so I may just rebuild it later into the same configuration and use it to backup the new one.


Ryzen woes
After the PC Case Gear payment portal decided it didn't like me being behind a corporate firewall I haven't placed my order. That turned out to be a good thing because there are a couple of major issues with them.

First the motherboard people are having to correct issues with their BIOS so most motherboards are hard to find at the moment.

Second the windows task scheduler doesn't understand they are hyper-threaded chips and so it moves tasks around more than it should effecting performance. That will probably require Microsoft to provide an update. How long and which versions of windows it will be available for is unknown. Linux (if you have an up to date kernel) doesn't have this issue.

Once these issues are ironed out I think they'll provided a great replacement for my 6 core/12 thread machines and I will place my order. I need to sell off my old hardware to make room for the new stuff.

26 February 2017

26th of February

Farm Status
Intel GPUs
All running Einstein Multi-directed Gravity Wave work

Nvida GPUs
Two running Seti work

Raspberry Pis
All except two running Einstein BRP4 work. The other two (a Pi2 and a Pi3) are running Seti Beta work


Seti multi-beam apps for Rpi
There has been some progress on the Raspberry Pi app which is why I have a Pi2 and Pi3 running Seti Beta. We're currently testing an updated app that is slightly faster than the previous one as well as being able to work on the ARM v6 (Pi Zero, B and B+). If you're going to run on a ARM v6 apart from it being very slow it also needs a patched kernel. Even the ARM v7 is slow when it gets a VLAR (Very Low Angle Range) work unit but at least it can do them now.


AMD Ryzen
I've asked my usual computer shop to quote on a couple of Ryzen S7 1700 machines to replace the two i7-5820K gas-guzzlers. The parts showed up on the PC Case Gear website yesterday (due for delivery 3/03/2017). I'm looking at two for the moment but given they use a bit less than half of the wattage of my current i7's I could get more AMD machines. I would expect they'll be slower than the i7 but then I get two extra threads so it will probably produce around the same throughput.


Storage Server
I am also looking at updating the file server, or at least its storage capacity. Currently I have an Intel RAID controller in it that provides 4 ports and there are 4 x 4TB drives in there.

SuperMicro have a storage server which is a similar motherboard to what I already have with a built-in SAS controller and another 10 SATA ports on the motherboard provided by an Intel chipset and it comes with 12 (3.5 inch) hot-plug drive bays on the front and another two (2.5 inch) on the back. The bad news is they is they only support Windows Server (2008 or 2012) or RHEL on it.

There are a number of other options such as a SAS expander, another RAID controller, bigger drives and so on that I could use which I will look into.


ZFS on Linux
The file server had me looking at ZFS on Linux. My first attempt was so try and install it on a Pi2 which failed miserably. It downloads the code and you have to compile it. The compile failed for a couple of reasons:
  • Its written with AMD64 architecture in mind
  • You have to get the Linux kernel headers to compile it
The Pi2 and Pi3 are currently ARM (32 bit) architecture. Technically the Pi3 could run ARM64 architecture as it an ARM v8 CPU but the Raspberry Pi foundation don't have it available.

As for the Linux kernel headers one has to get the version from the Raspberry Pi foundation that matches whatever version they are on (mine currently have 4.4.38). That isn't a simple apt-get command, you have to back-track the 4.4.38 version to the git check-in to find it.

My next attempt was to install Debian Stretch onto one of the i7-6700's which sort of worked. It gave some errors at bootup. I guess it doesn't have the necessary drivers for an H170-Pro motherboard. As well as that the beignet-opencl-icd didn't support the HD Graphics 530 that they have so I gave up on that one and reinstalled Win7 on it.

Ubuntu has a pre-compiled one in their repository so you don't have to compile your own, however there is some debate around the licensing of it.

28 January 2017

28th of January

Its still hot here in Sydney. We've had a few cool days where I have managed to run some Einstein or Seti work. The Einstein CPU tasks take about 16 hours so I am tending to run Seti as they're somewhere between 1 and 3 hours depending on the work unit.

I purchased another 5 port USB charger which I am using to power four Raspberry Pi2's. Its an Astrotek brand with a part number of AT-UPS-008B. Provided you only use 4 out of the 5 ports its fine. I have another one running four of the Pi3's. It might actually be able to run five (it reduces power output to each port when you plug in a 5th device) but I haven't tried five.

17 January 2017

17th of January

Its summer here in Sydney and its hot. Today was 32 degrees (C). The only things running on the farm are the Raspberry Pi's.

I got another 5 port USB charger. The ones I'm using to run the Pis are an Astrotek brand. Its a 5 port 8 amp unit. If you only plug 4 devices into it it can provide 2 amps per port. If you plug a 5th one in it drops the available power so I normally run four. The second charger is running the four Pi2's that have been sitting around gathering dust. I get a RAC (recent average credit) on Einstein BRP4 of about 300 from a Pi2's and 515 from a Pi3.

There are some rumours of a Pi4. There is nothing official at the moment but historically the Raspberry Pi foundation announce new models in February. The suggestion is it will have an ARM Cortex A53 processor at 1.8Ghz and 2Gb of memory. There are contrary suggestions that there won't be a new one for a few years as the foundation ordered 600,000 Pi3's and would need to sell these first.