15 October 2017

15th of October

Farm status
Intel GPUs
Running Einstein work overnight

Nvidia GPUs
Off most of the week

Raspberry Pis
Running Einstein work


Debian point release
After last weeks Debian 9.2 point release I have been trying different Nvidia drivers. The 375.82 drivers that the point release provided switch into low power mode after about 15 minutes on my GTX1060’s. I raised a bug with Debian and was told its an upstream (Nvidia) problem. There are a few versions of the 375.82 driver in the different repos but they all seem to have the problem. There is a 384.90 driver in experimental that doesn’t have the issue so I have installed it on all of the Nvidia GPU machines.

In addition to the Nvidia driver I have been reinstalling all the Intel GPU machines with the point release as it gets them off Jessie, now that the desktop issue has been resolved by an updated kernel. I had the Intel GPU machines running Einstein gravity wave work overnight.


BOINC testing
There is a BOINC client 7.8.3 out for testing. The BOINC Manager still has the jumping tasks tab/transfer tab issue, as well as the event log losing its date format when it scrolls. I never experienced the buffer overflow issue because I am not running the CPDN project at the moment. BOINC Manager issues aren’t much of a problem for me as I use BOINCtasks for managing the farm most of the time.

08 October 2017

8th of October

Farm status
Intel GPUs
Eight running Einstein gravity wave work

Nvidia GPUs
All off. Were running Seti overnight

Raspberry Pis
All ten running Einstein BRP4 work


Other farm news
I couldn't ssh into Pi #9 so I had to get it to finish off its work and reimage it. Strangely enough I could access it via BOINCtasks but ssh somehow got corrupted.

I hit the 50 million credits for Seti earlier in the week. The Intel GPU machines are now working on Einstein tasks.

The weather is going to be hot tomorrow so everything except the Raspberry Pis will be off again.


Stretch-backports on Rpi updated
While updating Pi #9 I followed the instructions I gave in September 2017 about adding Stretch-backports. They didn’t work. It seems the Debian gpg key signing has problems. To access backports requires the Wheezy and Jessie gpg keys (Debian 7 and 8). I have updated the instructions.


Debian Stretch 9.2
They did a point release to Stretch (9.2) yesterday. There is good news and bad news.

The good news is it has a kernel update to 4.9.0-4 (4.9.51) that allows me to upgrade the Intel GPU machines. ie It resolves the bug of having a blank desktop. I have already upgraded two of them.

The bad news is it has a bug in the Nvidia driver 375.82 which has now crippled my Nvidia GPU machines. I've raised bug 877971 with Debian about it. The driver switches into P8 (idle) mode after about 15-20 minutes of compute work. A reboot gets it back into P2 mode for another 15 minutes or so.

02 October 2017

1st of October

Farm status
Intel GPUs
Five running Seti work overnight. Three off.

Nvidia GPUs
Three running Seti work overnight. One off.

Raspberry Pis
All ten running Einstein BRP4 work


Debian donation
In my previous post I mentioned trying to help their build farm, specifically the armhf part. Attempts to get the Maxwell development boards they use have failed. The local Australian distributor didn’t respond to email or phone calls.

I have donated some money to the project. It will probably go into a general fund as there is no option to leave a comment even.


BOINC 7.8
I updated 4 of the Raspberry Pi’s by using the stretch-backports repo from Debian. See last months post about how to add stretch-backports to a Raspberry Pi.

Due to a couple of bugs which were introduced with 7.8.2 I won’t do the rest just yet. The BOINC release manager at the time didn’t include the fixes even though they were available. We’re hoping there will be a 7.8.3 to address the most pressing issue to do with cleaning out slot directories.


Other news
There are some details of the 8th generation i7’s being released. Looking at the i7-8700 compared to my i7-6700’s it would seem they are a little slower (3.2GHz compared to 3.4Ghz) but have two extra cores while sticking to the 65 watt TDP. There is also talk of a Ryzen 2 and 3 but no details yet.

As you can tell from the farm status the weather is starting to impact the farm. I haven’t quite reached 50 million credits for Seti yet but will resume the other projects once I reach it.

16 September 2017

16th of September

Farm Status
Intel GPUs
Five running Seti work, three off

Nvidia GPUs
Three running Seti work, one off

Raspberry Pis
All ten running Einstein work


Other news
There was a Bluetooth bug discovered that effects pretty much everything - Computers, phones car audio even. Microsoft patched Windows last month and the Linux guys had fixes out a couple of days ago. As a general rule I disable WiFi and Bluetooth in my computers when I install them. I prefer wired connections and not getting irradiated. Wired connections are also faster and reliable.


Debian donation
I was in contact with the Debian build farm team regarding the armhf architecture part of the farm. I was going to try and get a couple of machines together to donate to them. The seem to have a bunch of Marvell DB-78x00-BP based computers, but after they wanted them as rack mount that pretty much killed off the idea. Rack mounts are too big and heavy to ship around the world.

I did try and make contact with the local Australian Marvell sales representatives. According to the Marvell website they have one in Melbourne. I sent an email but haven't received any response from them. I may still make a financial donation to Debian instead of hardware so the build farm can be expanded. My reasoning is that they have a long backlog (it looks like 35 days) to build apps so the armhf versions come out much later than the other architectures.


Seti push
Weather-wise its been warm during the day so the Nvidia GPU machines have been running overnight only. I passed 48 million credits for Seti last week and decided everything can run Seti until I hit 50 million. This coincided with Asteroids having a week-long outage for maintenance work.

03 September 2017

3rd of September

Farm status
Intel GPUs
All off

Nvidia GPUs
All off

Raspberry Pis
All running Einstein BRP4 work


Farm news
Its supposed to be 28 degrees today so I have shutdown most of the farm apart from the Pis.

I got the replacement fans for the Fractal Designs case. The cable is shorter than the ones already in the case. I had to swap the rear fan to the front and put the new fan in the rear of the case so the cables can reach the fan headers on the motherboard. The top fan is rattling so its probably going to fail soon. I didn't replace it at the time because I would have to remove the motherboard or heatsink to get to the fan mounting screws.


Breakthrough Listen/UCB in the news
There was an article in the news about the Seti search. While it doesn't mention Seti@home it does mention Breakthrough Listen, UC Berkeley and the green bank telescope that we're processing data from.

Article: https://www.gizmodo.com.au/2017/09/more-than-a-dozen-new-cosmic-radio-bursts-detected-from-deep-space/


Add stretch-backports to a Rpi

If you want to use stretch-backports on a Raspberry Pi you can't easily because the Raspberry Pi foundation don't have a repository for it. You can, if you have a Pi2 or Pi3, use the standard Debian repository.

To add the Debian Stretch repository type the following commands into a terminal window or ssh into your Pi and type them in:

sudo apt install dirmngr
gpg --keyserver pgpkeys.mit.edu --recv-key  8B48AD6246925553
gpg -a --export 8B48AD6246925553 | sudo apt-key add -
gpg --keyserver pgpkeys.mit.edu --recv-key  7638D0442B90D010
gpg -a --export 7638D0442B90D010 | sudo apt-key add -



sudo nano /etc/apt/sources.list

Add a line like this:
deb http://deb.debian.org/debian/ stretch-backports main contrib non-free

Do a "sudo apt update" command to refresh the list of available packages.

To add a package that is in backports you can do a "sudo apt install -t stretch-backports xxx" where xxx is the name of the package you want to install.


27 August 2017

26th of August

Farm status
Intel GPUs
Seven running Einstein gravity wave work. One off

Nvidia GPUs
Two running Seti work. Two off.

Raspberry Pis
All ten running Einstein BRP4 work.


Power outage
When I came home on Friday I was greeted by a rather quiet computer room. The power had gone off around midday. While the power had come back on only the Raspberry Pi's had automatically restarted.

After restarting the PCs there doesn't appear to have been any damage apart from a couple of work units that got computation errors on one machine. The file server has a UPS so it shuts down automatically after a period of time.


Pi news
I have been updating the Raspberry Pi's to the latest release (Stretch). I reimage the SD card with the Stretch Lite image, boot up with a screen and keyboard attached to the Pi and run raspi-config. After that I can plug it into the network, install the boinc-client and ntp on them, tweak some configuration files and put a firewall on.

I have done all ten number crunchers and they are back doing Einstein BRP4 work. I still need to update the supporting Pi that has a Pi Drive. The Pi Drive means I have to move the root partition across before installing anything on it.

I read an article where they interviewed Eben Upton regarding the Pi3. Eben is the co-founder of the Raspberry Pi foundation. In the interview he is quoted as saying the Pi3 "feels like a 3 year device" suggesting we'll have to wait another 2 years before the next generation of Raspberry Pi becomes available.


Other news
Still waiting for 140mm Fractal fans to replace a failed one. The shop hasn't given me an ETA so I will need to chase them up next week. I am down one machine in the Intel GPU group.

The weather has been warm during the day so only two of the Nvidia GPU machines are running at the moment..

20 August 2017

20th of August

Farm status
Intel GPUs
Seven running Asteroids work.

Nvidia GPUs
Two running Seti work

Raspberry Pis
All running Einstein BRP work


Other news
One of the Intel GPU machines is off as its front case fan has died a rather noisey death. Its an older version of the Fractal Designs ARC Midi case. I have asked the computer shop to get me some replacement fans. I expect the other machines that are in the older cases will also fail in due course so having some spares is a good idea..

I also had an older 8 port Netgear switch where some of the ports stopped working. I replaced it with a newer one during the week. While I was at it I got a spare, just in case another switch stops working.


Software updates
I did install the 4.11 kernel onto the core i3 machine and it fixed the disappearing desktop issue with Debian Stretch on iGPUs. I had to do it via ssh seeing as I couldn't get a desktop after installing Stretch. Unfortunately if you use the 4.11 kernel there are issues with some apps that use vsyscall as its disabled by default. It can be enabled via a command-line argument on kernel start-up.

I have installed BOINC 7.8.1 on one of the Nvidia GPU machines in order to test it. The manager has a few issues and my drystone benchmark figure has doubled. The issues with the manager while annoying seem to have occurred some time after 7.2.42 and have been there for quite a while. The manager issues don't have any impact on BOINCtasks which is how I normally manage the farm. I don't expect they will be fixed before an official 7.8 BOINC is released.

The Raspberry Pi foundation have updated Raspbian to the Stretch release. Raspbian for those that don't know is Debian but with some customisation for the Pi. They recommend you reinstall it rather than upgrading, but you can upgrade an existing Pi if you wish.

13 August 2017

13th of August

Farm status
Intel GPUs
Have been running Asteroids and Einstein work. Currently idle.

Nvidia GPUs
Have been running Seti work. Currently idle.

Raspberry Pis
Running Einstein BRP4 work


Other news
I tried installing Stretch on one of the Intel i3's and had to go back to Jessie to get the desktop back. I thought installing Beignet (the OpenCL package for Intel GPUs) might help the desktop work under Stretch but it doesn't make any difference. There haven't been any updates to the Debian bug that I raised regarding the desktop issues with Intel iGPU's and Stretch. There is a later kernel (4.11) available from Stretch-backports which I have yet to try. Stretch comes with a 4.9 kernel.

The weather has warmed up during the day even though its still winter. The farm is idle during the day and crunching overnight now.

I did software updates to all Raspberry Pis today. After rebooting Pi #5 refused to come back so I had to reimage the SD card and install it again. It cancelled the 5 work units that it had. If Einstein had "resend lost tasks" enabled on their server I could have downloaded them again but it seems they don't have it enabled at the moment.

06 August 2017

6th of August

Farm status
Intel GPUs
All running Einstein gravity wave work

Nvidia GPUs
Two running Seti work

Raspberry Pis
All running Einstein BRP4 work


HT Condor
I'm experimenting with HT Condor at the moment. The HT stands for high throughput. It a batch task queuing system that a lot of computer clusters run. I am using one of the i3's for experimenting. The idea is to be able to run work via HT Condor or BOINC. With HT Condor you have a submit node, a manager node and one or more compute nodes. They can all be the same machine or they could be separate machines depending on the size of the cluster. Its available as a Debian package so its easy to install and get going.


Other news
While using one of the i3's with its built-in graphics I thought I would try Debian Stretch on it. It did the same as my i7's and doesn't display the desktop icons after logging in. All I get is the Debian wallpaper and a mouse pointer. I did have the i3's with a GTX970 running Stretch without any problems when using the Nvidia drivers. It just doesn't seem to work with the built-in Intel graphics. I have updated the bug I raised with Debian.

I haven't tried running all four of the Ryzen's with GPU tasks at the same time. I'm not too sure if the power circuit will be able to handle all of them running at once. I have run all of them doing just CPU work which is fine. I have run 3 of them doing both CPU and GPU work. I need to plug the power meter in which means rearranging power cables and I need to buy new batteries for the meter (it uses the little button ones that don't last very long).

30 July 2017

30th of July

Farm status
Intel GPUs
All running Einstein gravity wave work

Nvidia GPUs
All off

Raspberry Pis
All running Einstein BRP4 work


Other news
Debian updated their net install image for Stretch to 9.1 so I tried installing it on one of the Intel GPU machines. Unfortunately no desktop so I had to reinstall Jessie again to get it going. I updated the Debian bug but there doesn't seem to be any progress on it.

We had an unusually warm weekend so the Nvidia GPU machines were all switched off. Its meant to be the middle of winter at the moment.

Project-wise Asteroids got up to 41.4M credits so now Einstein is playing catch up and is currently on 41.2M. Seti is on 44M credits but I have only been running it on two of the Nvidia GPU machines.

I still need to look into running CPDN work on the Linux machines. Unfortunately their apps are only 32 bit and that means various 32 bit libraries need to be installed. You would think the project would switch to 64 bit for Linux at least but they don't seem inclined.

25 July 2017

25th of July

Farm status
Intel GPUs
All running Asteroids work

Nvidia GPUs
Two running Asteroids work

Raspberry Pis
All running Einstein BRP4 work


Other news
I have the other two Ryzen systems installed and running. The Nvidia GPU part of the farm is now 4 x Ryzen 1700's with a GTX1060 in each one.

When I started up the last two machines they gave me a "CPU fan error" which turned out to be the shop had plugged the CPU and case fans into the wrong headers on the motherboard. I had to plug them into the correct ones. I also took the opportunity to remove the DVD drive from each machine. In the dim, dark days when I ran windows I needed them to install it, but these days I boot off a USB memory stick and install Linux from it.

I have pretty much reached the power and thermal limits of the farm in its current location so that puts any further expansion on hold until I can find somewhere else to locate equipment. I will look at adding some redundancy (internet, etc).


BOINC news
There are some moves afoot to improve the BOINC client. At the moment its just a list of what we'd like to see. Its may not actually happen but you have to start somewhere. There are also moves to come up with better source/release management of the client

16 July 2017

16th of July

Farm status
Intel GPUs
All running Asteroids work.

Nvidia GPUs
Two AMD Ryzen's running Seti work.

Raspberry Pis
All running Einstein BRP4 work


Other news
The i3's and two new Ryzen's were dropped off at the computer shop last week. They have done the musical cases bit and installed the hardware, but they aren't open on weekends any more so I have to take some time off next week to go and collect them.

Asteroids ran out of work mid-week so I switched the Intel GPU machines to Einstein, except they were doing some server reconfiguring so I couldn't get any work from them either. I ended up running Seti for a day before picking up some Einstein gravity wave work. Since then Asteroids have put more work up. Asteroids is on 39M credits and Einstein is on 40.4M credits at the moment.

I don't currently have any plans for more Ryzen's, however I did have a brief thought of replacing the 8 i7-6700's with 4 x Ryzen 1700's. They'd use half the electricity (due to reduced numbers of machines). They are both rated at 65 watts. The i7's are of course faster than the Ryzen. I would also need to get a graphics card for each of the Ryzen's as they don't have built-in graphics.


Raspberry Pi news
I was contacted by an American user asking about running BOINC on a Raspberry Pi cluster that he had. The short answer was yes it can be done but its not cluster-aware so you have to run an instance of BOINC on each node. If the nodes are Pi2's or Pi3's then it recognises them as a quad-code CPU and will run 4 tasks at a time.

As for projects there is Asteroids, Seti and Einstein that I have run. Asteroids is very slow (they increased work unit size but didn't optimise their app for the Pi2 or Pi3). Seti is also slow but has been optimised to use whatever features the Pi has available and so has the Einstein app. There are others but haven't tried them. I tend to run Einstein on my Pi3's, but you need to select "run beta apps" on your project preferences on the Einstein website to get the optimised app.

On that note there was an article on the Rpi blog about OctaPi and how to create a Pi cluster. See https://www.raspberrypi.org/blog/octapi/ for the details. Now if we could get BOINC to run on the head node acting as the scheduler and use MPI (message passing interface) to communicate with the compute nodes...

10 July 2017

9th of July

Farm status
Intel GPUs
Running Einstein gravity waves during the week. All doing Asteroids work now.

Nvidia GPUs
Two Ryzens are running Seti work. Two i3's finishing off Einstein gravity wave work.

Raspberry Pis
All running Einstein BRP4 work.


Other news
The parts for the next two Ryzen builds arrived during the week. I need to drop them off at the computer shop. They will assemble them and move the i3's into Fractal Designs ARC Mini cases that are currently empty. This will bring the farm up to 4 Ryzens with a GTX1060 in each. That's 32 cores/64 threads in the Ryzens and 32 cores/64 threads in the i7's.

I haven't been running Asteroids work recently so I have all the Intel GPU machines currently running it. Asteroids is on 37M credits. I passed 40M credits for Einstein during the week so I am letting Asteroids catch up. Seti is on 42.6M credits.


Debian bugs
I haven't heard anything back regarding the two Debian bugs I have raised.

The Mate desktop doesn't seem to come up after doing a clean-install of Stretch on the Intel GPU machines. I had to go back to Jessie to get it working. I was asked for a list of files that were installed and have provided that.

The second bug is the all_projects_list.xml file is missing from the current boinc-client package which results in being unable to attach to projects. A work around is to copy the file from another computer if you have it. There was some discussion on the boinc-dev mailing list regarding this being a bug in the core client as it is supposed to download it if the file is missing but doesn't. There is some work on a new boinc-client with a couple of bug fixes and a framework for some new features so it may get fixed.

01 July 2017

1st of July

Farm status
Intel GPUs
Two running Seti. One off. Remaining 5 running Einstein gravity wave work.

Nvidia GPUs
Two Ryzen's running Seti. Two i3's running Einstein gravity wave work.

Raspberry Pis
Ten running Einstein BRP4 work.


Other news
I have ordered another pair of Ryzen's, motherboards and memory. These will replace the i3's.

I mentioned last week that I tried upgrading one of the Intel GPU machines from Debian Jessie to Debian Stretch. It won't display the desktop when I log in. I've raised it as a Debian bug. They asked for some more information so I had to reinstall Stretch and give them that. It is switched off but I left Stretch on it in case they ask for further information. If I haven't heard from them by tomorrow I will reinstall Jessie so I can get it crunching again.

The quarterly electricity bill arrived last week. It was $800. The computers are responsible for almost all of it. The higher prices start today.

25 June 2017

25th of June

Farm status
Intel GPUs
Half running Einstein gravity waves and the other half running Seti work.

Nvidia GPUs
The two Ryzen's are running Seti and the two i3's are running Einstein gravity waves.

Raspberry Pis
All running Einstein BRP4 work.


Rpi news
I finished reconfiguring the Pis. There are now 10 Pi3's crunching and one more provides support. Five of the crunchers have a 40mm fan bolted on top of their case for cooling. The other five have a desk fan to provide air flow.


Stretch news
I installed Debian Stretch a number of times during the week to see how things go. I had to reinstall once after the machine I was upgrading from Jessie triggered the screen saver and wouldn't come back. A subsequent reinstall moving the mouse every 5 minutes worked fine.

I also found out the hard way that rc.local doesn't run any more as one of my start-up scripts wasn't getting run. I ended up having to define a service to run it. I also had to set execute permissions on the script and service otherwise they don't run. They used to work fine under Jessie.

I tried upgrading one of the Intel GPU machines however when I logon instead of getting a desktop there is just the wallpaper and a mouse pointer. No icons, no taskbar and no menu bar. I tried reinstalling it twice with the same result so I've raised a Debian bug for that one.

I clean-installed the Nvidia GPU machines. They are all running Stretch now. If you are clean-installing be aware the enumeration of network ports is changed, they are no longer called eth0, eth1 any more.

17 June 2017

17th of June

Farm status
Intel GPUs
All running Seti work

Nvidia GPUs
All running Einstein gravity wave work

Raspberry Pis
All running Einstein BRP4 work


Rpi news
I got 3 more Pi3's. One replaced the faulty #9. I added fans to the tops of a couple of the official Pi cases. Unfortunately I didn't get the hole centered on one of them so that is one case messed up. A second Pi3 (with fan) has been installed and is now crunching away. I will add the other new one into the mix which will bring the bramble up to 10 Pi3's crunching.

The Pi2's have been retired. The bramble is close to 2 million credits of Einstein crunching.


Other news
Debian Stretch is being released as I write this. I will wait a week or two before looking at upgrading crunchers. Jessie is still supported for another year. It may be better to clean install Stretch rather than upgrading them from Jessie.

I tried the GPS receiver on a couple of machines, one Intel and a Raspberry Pi, it would happily talk to the computer but it seems it couldn't get a fix on the satellites. I had it stuck to the inside of a skylight window. Its not weatherproof so I can't just hang it outside. I will try it outside for a short while to see if I can get it to work.

10 June 2017

10th of June

Farm status
Intel GPUs
Running Einstein gravity wave and Seti work

Nvidia GPUs
The Ryzens are running Seti work. The i3's are running Einstein gravity wave work

Raspberry Pis
All running Einstein BRP4 work


Rpi news
It seems there is a Trojan out that specifically targets the Raspberry Pi. Apparently a lot of people don't change the default password. Here is a link to the article: http://www.guru3d.com/news-story/new-linux-trojans-installs-crypto-currency-mining-software-on-raspberry-pi.html

The copper heatsinks from Enzotech arrived yesterday. I have ordered 3 more Pi3's and a couple of cases.


Electricity price increase
My current electricity retailer has announced a 16% price increase for electricity and 9% for gas from the 1st of July. There are a couple of other retailers I could switch to. I will compare them but I expect they will all increase around the same time.

The electricity price increase would mean the Ryzen upgrade (from a 1700 to 1700X) would not be cost effective. It uses another 30 watts for around a 10-12% increase in output. I might have made the right decision about which Ryzen chip to get after all :-)

I have seen a natural gas powered generator around the size of a washing machine for business use but they have a high purchase price. A diesel or petrol powered generator is going to be cheap to buy but cost a lot to run. Its time to look into a solar and battery solution.


Other farm news
My current planning has me looking at another pair of Ryzen 1700's. I have spare cases, hard disk drives and graphics cards for them to keep the cost down. This would free up the i3's for something else although I don't currently have anything in mind for them.

04 June 2017

4th of June

Farm Status
Intel GPUs
All running Seti work

Nvidia GPUs
All running Seti work (after running Einstein gravity waves during the week)

Raspberry Pis
All running Einstein BRP4 wotk


Other news
Last weekend the farm was off, except the Raspberry Pis, due to having to attend an out of town wedding. The Ryzens and the storage server were delivered the day prior to the wedding. Unfortunately I didn't have time to configure them before leaving.

The first Ryzen was setup last week and spent the week running different projects before switching to Seti work. The second machine was setup yesterday and is also doing a range of projects. Performance is a bit slower than the Intel i7-5820K's that they replaced. I am considering replacing the Ryzen 1700 with 1700X CPUs (3.4GHz base clock) but have to weigh up the extra power consumption (another 30 watts). I will also need to replace the heatsink but can reuse the AM4 mounting kit. Each Ryzen also has a GTX1060 installed.

The firmware still hasn't caught up with the Ryzens and Linux throws a number of errors when starting up even though I have a 4.9 kernel installed and the latest firmware from ASUS. I expect they will need a few updates before most of the bugs are fixed.

I've got another batch of 14mm copper heatsinks on the way from Enzotech, so I can look at replacing the two Raspberry Pis mentioned in my last blog post. I still need to look at a better case for multiple Pis.

20 May 2017

20th of May

Farm status
Intel GPUs
Spent all of last week running Einstein gravity wave work. Now back to Seti

Nvidia GPUs
Running Einstein gravity wave work (CPU only)

Raspberry Pis
Running Einstein BRP4 work


Other news
Now I have the second internet connection I split the network into two. The Raspberry Pis are now over on the 2nd net. It took a while to unhook the Pis from the main network with network cabling and having to dig out an unused switch. Software-wise I spent a bit of time removing dependencies on machines that are now on the main network and therefore inaccessible. Two of the Pis have Pi Drives (USB hard disks) so one of them was turned into a proxy and time server.

One problem I have with the Pi's is they don't seem to broadcast their device name so I can only reference them by their IP address, which means they have to be assigned a static IP address. Sure they can get a dynamic IP address from the router but I can't access them by name, only by their IP address. The PC's running Linux don't have this problem so it seems something unique to Raspbian.

I will order a couple more Pi3's. I need to replace a faulty Pi3 and replace the one that got used for the proxy/time server. I also need to order more copper heatsinks from Enzotech.

One of the other projects I am doing is a more accurate time server for the main network. I have purchased a USB GPS receiver which can be used to obtain accurate time. It needs a view of the sky and the machine I am intending to connect it to is furthest away from the window/skylight. Today I have ordered a USB extention cable so I can connect it.

The pair of GTX1060's that I ordered for the Ryzen machines arrived during the week. Unfortunately the Ryzen's still haven't been built due to delays with getting the heatsinks. Maybe next week.

13 May 2017

13th of May

Farm status
Intel GPUs
All doing Einstein Gravity wave

Nvidia GPUs
Doing Einstein Gravity wave (CPU) plus Seti GPU work

Raspberry Pis
Doing Einstein BRP4 work


Project news Einstein
They had a problem with their first tuning run so we've got updated apps and are doing a second tuning run.


Other news
Still waiting on the Ryzen builds. The computer shop were waiting on Noctua heatsinks to come in. It would have been nice to see how they perform on the Gravity wave tuning run as its CPU only. Hopefully they'll be ready next week.

The new Drobo arrived. I had a bit of a problem with it due to taking the mSata SSD from the old Drobo. A software reset fixed that. I think its 50% faster than the old one. I need to list the old one on eBay now that I've cleaned off the hard disks.

The second ADSL connection is running however I need to work out how to load balance the two internet connections. I know TP Link make a $130 load balancer device which might be how I'll run it. I need to switch the main internet connection back to a standard ADSL first. It seems the wiring between the communications cupboard and the house isn't too good because they can get 18Mbit to the comms cupboard, but to the house only gets 13Mbit.

07 May 2017

7th of May

Farm status
Intel GPUs
Mostly Seti with a batch of Einstein work

Nvidia GPUs
Mostly Seti with a batch of Einstein work

Raspberry Pis
All running Einstein BRP4 work


Project news Einstein
They've started another run of their gravity wave work units. Currently its a tuning run to get an idea how long each type of CPU takes and iron out any bugs with the apps. There are two types of work unit called Hi and Lo. When I ran all 8 cores on the i7-6700's the Hi ones were taking 12-13.5 hours. When running half the cores they were taking 9 hours.

My i7-6700's only have 16Gb in total. The Ryzen's will have 32Gb. The higher frequency work units need 2Gb of memory each so I can only run 7 on the i7's and 14 on the Ryzen's.

The Lo work units are sent to slower machines and don't take as long. The i3-6100T's were talking around 3.5 hours when running all cores. These work units also use much less memory (around 270Mb).


Other news
The two AMD Ryzen 7 motherboards, CPUs and memory kits arrived and I dropped them off at my usual computer shop. I supplied the GPUs and hard disks and they are supplying the case, power supply, cooler and assembling it. Hopefully they will be ready next week.

The two i7-5820K's were sold off and have gone. That leaves the farm with two working GPU's at the moment. I kept the GTX970's which will go into the Ryzen's to start with and get swapped for the GTX1060's when they arrive, supposedly on the 18th.

The Drobo 5N2 is due to arrive next week some time.

30 April 2017

30th of April

Farm Status
Intel GPUs
All running Seti work

Nvidia GPUs
Running Seti work

Raspberry Pis
Running Einstein BRP4 work


Transition to Linux
The last two machines were converted over to Linux without any issues. I simply follow the process I documented before. That means there are no Windows crunchers in the farm now.


Other news
The same person that purchased all the motherboards also wants to buy the i7-5820K machines. I decommissioned them and removed the GTX1060's on Friday. I will be replacing them with a pair of AMD Ryzen 1700 machines. The GTX1060's will go into the Ryzen's when I get them.

I have also ordered another pair of GTX1060's. These will initially be used to replace the GTX970's. Longer term I will probably replace the i3's with another pair of Ryzen machines.

My original idea was to have a low powered CPU (the i3-6100T is only 35 watts) and a pair of high performance GPU's. Unfortunately the i3 doesn't have enough PCIe lanes  to run two GPU's at x16 speed. It seems to provide one slot at x16 and the second at x4 and this is on a Z170 chipset motherboard. The i3 does have built-in graphics and the motherboard is mATX sized so I may be able to use them for some other purpose.

23 April 2017

23rd of April

Farm Status
Intel GPUs
All running Seti overnight

Nvidia GPUs
Two running Seti overnight. Other two are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
The WD Red hard disks turned up. I expect the Drobo that they are destined for will arrive late next week.

I also sold off a couple more GPU's and the last remaining motherboard.

Now that I have sold off  most of the excess hardware I will look at replacing the i7-5820K machines (6 core/12 thread) with AMD Ryzen's. I'm looking at the Ryzen 1700 (3Ghz) but also only 65 watts. The motherboard would be an ASUS Prime X370 with 32GB of DDR4 memory. I will keep the GTX1060's that are in there as they are the most up to date GPU's that I have.


Transition to Linux
In the blog post prior to this one I covered the steps used to converted two of my Nvidia GPU machines to Linux. They can run CUDA, OpenCL and CPU work now. I have two more to convert and that is all the farm converted to Linux apart from two windows-based machines. One is the file server and the other is a laptop that I use to monitor the farm using BOINCtasks.

16 April 2017

Setting up a Linux CUDA cruncher

Rather than using an outdated Ubuntu which seems to be the current suggestions, this is what I used with Debian. This will get you the latest Debian (Jessie), latest kernel (4.9) and the 7.6.33 BOINC client. You should be up and running with a CUDA and OpenCL capable machine after doing this.


Part 1 - Install Debian
I used the Debian 8.7 net install for this. You’ll need a thumb drive or a blank CD. Download Debian from http://www.debian.org/distrib/ and write the ISO image to CD or thumb drive.

Boot off the thumb drive or CD. It will start up the Debian installer

Install Debian. It will ask a bunch of questions like language, country, etc and prompt for partitioning your disks. When it asks for software to install select SSH server and whatever desktop you prefer and remove all other selections. Once done it will prompt for a reboot.


Part 2 - Install Nvidia software
Login as root, open a xterm window and type the following commands:

cd /etc/apt

nano sources.list (nano is a text editor)

Change “jessie main” lines to “jessie main contrib non-free” and add a jessie-backports line. It should look like this when you're done. I'm using httpredir as it will pick the fastest server.

deb http://httpredir.debian.org/debian/ jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-updates main contrib non-free
deb http://httpredir.debian.org/debian/ jessie-backports main contrib non-free

Exit out of nano and save the file (Control-O followed by Control-X)

apt update

apt install –t jessie-backports firmware-realtek (if needed). Most of my motherboards have Realtek chips.

apt install –t jessie-backports linux-image-amd64

apt install –t jessie-backports nvidia-kernel-dkms nvidia-smi nvidia-xconfig

apt install –t jessie-backports nvidia-opencl-icd (if you want OpenCL support)

nvidia-xconfig

  
Part 3 – Install BOINC
apt install –t jessie-backports boinc-nvidia-cuda boinc-manager

sync

reboot

14 April 2017

Good Friday 2017

Farm status
Intel GPUs
All running Seti work

Nvidia GPUs
Did a couple of GPUgrid work units earlier in the week. Currently they are off.

Raspberry Pis
All running Einstein BRP4 work


Other news
I have ordered three 4TB WD Red hard disks for the Drobo 5N2. The Drobo isn't due here until the end of the month. The price per terabyte increases with the larger capacity drives. Two 4TB drives are still cheaper than one 8TB drive. Seagate have a 12TB drive but they cost over a thousand dollars.

A couple of the Pi's were playing up so I had to re-image them. Unfortunately when attaching them to the Einstein project they decided the program executable was corrupted and then trashed their work units. A project reset in BOINC fixed the corrupt program. It did this on both of them so I think the checksum value given for the executable is probably wrong rather than a download issue.

I've rescheduled the ADSL connection for the 26th of April so hopefully it will get installed this time. The last technician didn't have a key to the communications cupboard.


Linux transition
The Intel GPU's are doing work faster than when they were windows machines. All 8 of them are churning through Seti work. I am trying to get the credits similar for the 3 projects they've been set up to run (Asteroids, Einstein and Seti).

One of things to try on the transition to Linux is getting the Nvidia GPU's working. Debian have Nvidia drivers in their repository so hopefully they will work. Most of the guides I have seen install older versions of Ubuntu (14.04) or Mint which is based upon Ubuntu.

I also want to look at running CPDN jobs however their programs are all 32 bit so that usually causes problems when running on a 64 bit Linux platform.

08 April 2017

8th of April

Farm status
Intel GPUs
3 running Seti work. The rest are off.

Nvidia GPUs
All off

Raspberry Pis
All running Einstein BRP4 work


Other news
I sold off one of the ASUS P6T's. The other one and a GTX670 are on eBay ending tomorrow. I also sold off some Seasonic power supplies. I have interest in the remaining motherboards and a couple of GTX750Ti cards. That still leaves a bunch of other parts and the two i7-5820K system to sell off.

I had a phone company technician come out to install a second ADSL line. Unfortunately he didn't have a key to the communications cupboard. Its meant to be a standard set for phone companies but it seems they don't come prepared.

I'll put up a few more graphics cards on eBay next week to get rid of them.

I've ordered a new Drobo 5N2 to replace my Drobo 5N with. I need to order some larger WD Red drives to go in it so it can backup the storage server. Once its up and running I will sell off the 5N.

02 April 2017

2nd of April

Farm Status
Intel GPUs
All except one currently running Seti work. They were running Asteroids work before.

Nvidia GPUs
All off.

Raspberry Pis
All running Einstein BRP4 work


Farm news
I've changed all except one of the Intel GPUs over to Linux. That's 7 of them. The last one had an SSD + HDD which I had to swap out with the proxy servers SSHD. Its now the same configuration as all the other Intel GPU machines and currently installing Linux. I had to rebuild the proxy server.

I found that installing Beignet (Intels OpenCL drivers for Linux) doesn't work if you are on the 3.16 kernel. It will install but clinfo is unable to detect the device. Updating the kernel to a 4.x one seems to have resolved that. I can now use the iGPU but it slows down the CPU so is not normally worth the effort.

For the moment I am leaving the Nvidia GPU machines running Windows but plan on selling off the two GTX970's and also the two 6 core/12 thread machines which currently have GTX1060's. I'm going to keep the GTX1060's but will need to put something else in there in order to sell them. I will need to investigate running the GPUs under Linux and what driver combinations work.

The computer shop rang regarding the storage server. They have received it from their supplier but it seems to only have SAS expanders for each row of drive bays. Its got 12 drive bays but each set of 3 has its own cabling. The hard disk controller has only a single SAS connector meaning it can only run 3 drives. I'll have to chase them up next week. It seems the shop don't have experience with storage servers.

19 March 2017

19th of March

Farm Status
Intel GPUs
All off at the moment. They have been running overnight.

Nvidia GPUs
All off at the moment. The two GTX1060's were run overnight.

Raspberry Pis
All running Einstein BRP4 work


Linux testing
I did some more testing with Linux giving various VM's a workout. I have a Debian Jessie VM, a Debian Stretch VM and a Mint VM. I'm in a position to switch the Intel GPUs over to Linux without too much drama. About the only problem I see is each machine will get a new identity and lose its history of work done.

It seems Microsoft has started giving messages about unsupported CPU's on earlier versions of Windows with the latest CPU's which includes the AMD Ryzen. That means all new equipment I buy will probably be running Linux. Not the smartest idea from Redmond. It will be interesting to see how much market share they lose.


Storage Server
It is still on its way. I have already picked up a bunch of new 4TB hard disk drives to go in it. I have also ordered a 10Gbit network card to put in it. That means I'll have to get a couple of switches that support 10Gbit before I can make use of it. I am thinking of a 1Gbit switch with a 10Gbit uplink for most of the crunchers and a 10Gbit main switch. All my machines (apart from the Raspberry Pis and the AlphaServer) have 1Gbit network ports, some have two.

I need to make some room for the new server. I'm expecting to get rid of a number of CM Storm Sniper cases. Two of them still have ASUS P6T's and Corsair H80 water cooling in them, the other parts have been cannibalised. The other two are empty.

After that I need to sell off the existing i7-5820K's (I have two) before I can order some AMD Ryzen's to replace them with. Then there are all the old graphics cards that I need to get rid of...

07 March 2017

7th of March

Farm Status
Intel GPUs
All except one running Seti work

Nvidia GPUs
Ran all of last weekend. Currently off.

Raspberry Pis
All except two running Einsten BRP4 work


Linux testing
In my hunt to find alternatives to Windows I have been installing various Linux flavours on a couple of machines. My current leaning is towards Debian. I'd like to get an up to date kernel and they have an up-coming release called Stretch. When exactly it will be available is a guess, but maybe April 2017. Why do I need the later kernel, well those AMD Ryzen machines need an up to date kernel to correctly run tasks on them.

I spent a lot of time last weekend and again this weekend installing and re-installing to find out that either upgrading to Stretch or installing via the Stretch net-install (RC2) doesn't work and you lose the desktop.

At the moment I've setup a virtual machine and I'm on the 2nd reinstall of Debian after it decided to remove my entire desktop due to doing apt-get remove gstreamer* and apt-get autoremove commands.

I did have a play with Mint 18.1 (Serena) which is visually great however I don't want a full-blown desktop system just to run on the number crunchers and possibly storage servers. A light-weight desktop is enough for my needs. I can even work with no desktop and use the command-line if needed.


Storage server
I have ordered a bunch of 4TB hard disks and a 2U 12 bay storage server. Its taken almost 2 weeks of back and forth with the shop to get it configured the way I want.


I currently have a Windows-based file server with a RAID controller and 4 x 4TB hard disks and while it works fine expanding its capacity is rather difficult. As it turns out the one I am buying uses the same motherboard as my existing file server, so I may just rebuild it later into the same configuration and use it to backup the new one.


Ryzen woes
After the PC Case Gear payment portal decided it didn't like me being behind a corporate firewall I haven't placed my order. That turned out to be a good thing because there are a couple of major issues with them.

First the motherboard people are having to correct issues with their BIOS so most motherboards are hard to find at the moment.

Second the windows task scheduler doesn't understand they are hyper-threaded chips and so it moves tasks around more than it should effecting performance. That will probably require Microsoft to provide an update. How long and which versions of windows it will be available for is unknown. Linux (if you have an up to date kernel) doesn't have this issue.

Once these issues are ironed out I think they'll provided a great replacement for my 6 core/12 thread machines and I will place my order. I need to sell off my old hardware to make room for the new stuff.

26 February 2017

26th of February

Farm Status
Intel GPUs
All running Einstein Multi-directed Gravity Wave work

Nvida GPUs
Two running Seti work

Raspberry Pis
All except two running Einstein BRP4 work. The other two (a Pi2 and a Pi3) are running Seti Beta work


Seti multi-beam apps for Rpi
There has been some progress on the Raspberry Pi app which is why I have a Pi2 and Pi3 running Seti Beta. We're currently testing an updated app that is slightly faster than the previous one as well as being able to work on the ARM v6 (Pi Zero, B and B+). If you're going to run on a ARM v6 apart from it being very slow it also needs a patched kernel. Even the ARM v7 is slow when it gets a VLAR (Very Low Angle Range) work unit but at least it can do them now.


AMD Ryzen
I've asked my usual computer shop to quote on a couple of Ryzen S7 1700 machines to replace the two i7-5820K gas-guzzlers. The parts showed up on the PC Case Gear website yesterday (due for delivery 3/03/2017). I'm looking at two for the moment but given they use a bit less than half of the wattage of my current i7's I could get more AMD machines. I would expect they'll be slower than the i7 but then I get two extra threads so it will probably produce around the same throughput.


Storage Server
I am also looking at updating the file server, or at least its storage capacity. Currently I have an Intel RAID controller in it that provides 4 ports and there are 4 x 4TB drives in there.

SuperMicro have a storage server which is a similar motherboard to what I already have with a built-in SAS controller and another 10 SATA ports on the motherboard provided by an Intel chipset and it comes with 12 (3.5 inch) hot-plug drive bays on the front and another two (2.5 inch) on the back. The bad news is they is they only support Windows Server (2008 or 2012) or RHEL on it.

There are a number of other options such as a SAS expander, another RAID controller, bigger drives and so on that I could use which I will look into.


ZFS on Linux
The file server had me looking at ZFS on Linux. My first attempt was so try and install it on a Pi2 which failed miserably. It downloads the code and you have to compile it. The compile failed for a couple of reasons:
  • Its written with AMD64 architecture in mind
  • You have to get the Linux kernel headers to compile it
The Pi2 and Pi3 are currently ARM (32 bit) architecture. Technically the Pi3 could run ARM64 architecture as it an ARM v8 CPU but the Raspberry Pi foundation don't have it available.

As for the Linux kernel headers one has to get the version from the Raspberry Pi foundation that matches whatever version they are on (mine currently have 4.4.38). That isn't a simple apt-get command, you have to back-track the 4.4.38 version to the git check-in to find it.

My next attempt was to install Debian Stretch onto one of the i7-6700's which sort of worked. It gave some errors at bootup. I guess it doesn't have the necessary drivers for an H170-Pro motherboard. As well as that the beignet-opencl-icd didn't support the HD Graphics 530 that they have so I gave up on that one and reinstalled Win7 on it.

Ubuntu has a pre-compiled one in their repository so you don't have to compile your own, however there is some debate around the licensing of it.

28 January 2017

28th of January

Its still hot here in Sydney. We've had a few cool days where I have managed to run some Einstein or Seti work. The Einstein CPU tasks take about 16 hours so I am tending to run Seti as they're somewhere between 1 and 3 hours depending on the work unit.

I purchased another 5 port USB charger which I am using to power four Raspberry Pi2's. Its an Astrotek brand with a part number of AT-UPS-008B. Provided you only use 4 out of the 5 ports its fine. I have another one running four of the Pi3's. It might actually be able to run five (it reduces power output to each port when you plug in a 5th device) but I haven't tried five.

17 January 2017

17th of January

Its summer here in Sydney and its hot. Today was 32 degrees (C). The only things running on the farm are the Raspberry Pi's.

I got another 5 port USB charger. The ones I'm using to run the Pis are an Astrotek brand. Its a 5 port 8 amp unit. If you only plug 4 devices into it it can provide 2 amps per port. If you plug a 5th one in it drops the available power so I normally run four. The second charger is running the four Pi2's that have been sitting around gathering dust. I get a RAC (recent average credit) on Einstein BRP4 of about 300 from a Pi2's and 515 from a Pi3.

There are some rumours of a Pi4. There is nothing official at the moment but historically the Raspberry Pi foundation announce new models in February. The suggestion is it will have an ARM Cortex A53 processor at 1.8Ghz and 2Gb of memory. There are contrary suggestions that there won't be a new one for a few years as the foundation ordered 600,000 Pi3's and would need to sell these first.