Wednesday, 16 March 2016

Yintersync.NET development update

Things have been a little quiet recently because we are working on some new features.
  • Much faster initial/seed transfer.
  • Exclude files support.
  • Event log in email trimmed to the last 24 hours.
Hopefully we should have a new version within the next 6 weeks with some of these new features.

Tuesday, 8 March 2016

Windows Server 2012 R2 Data Deduplication in Yintersync.NET Hyper-V backup environment.

We wanted to test how well Windows 2012 R2 data deduplication worked with our set of 83 VHD's totaling 7.09TB. We did not want to test on our production disaster recovery setup so we needed a separate test environment.

We installed an extra 4TB Western Digital Red Pro in our existing Server 2012 R2 DR Server and copied on the first 3.5TB of Virtual Machines. We left the system to dedup overnight and copied more virtual machines on. After a few cycles of this our 7.09TB of VHD's fitted onto the 4TB drive using just 3.07TB. of space.

Windows performs deduplication when the system is not busy. The "Chunks" are taken out of the file and and placed in the System Volume Information\Dedup\ChunkStore. Files on the hard drive are now links to the relevant data within the chunkstore. Because of this, you can easily see when the deduplication is complete by the fact that a 200GB file is using "0 bytes" on the disk outside of the ChunkStore.

We next ran our first synchronization over the top of the deduplicated data. The files grew a small amount bigger than "0 bytes". Soon after the sync was complete, the deduplication started again.

Once we starting loading the disk up with lots of simultaneous synchronization, we noticed a significant performance drop when hashing data deduplicated files. With deduplication on a large mechanical disk the capacity to read performance ratio is very poor so the synchronization was unable to complete on our test set of data.

This test failed because we had set unreasonable performance expectations. Even at 130MB/s which is about as fast as the test disk runs in ideal circumstances, hash checking all of this data would take at least 16 hours. We were seeing disk speeds of around 40MB/s, far too slow. In the future we will continue testing Data Deduplication on a RAID 10.

Tuesday, 23 February 2016

Fast, cheap, off site and easy Exchange backup and disaster recovery using Yintersync.NET and Kernel for Exchange

Before we started creating Yintersync.NET one of our customers Exchange Server failed.

The backups were stored in a proprietary database which took around 30 hours to extract, the first attempt of which failed while we were sleeping. The users had to go without an Exchange Server for more than 2 days before we finally got things up and running again. Perhaps we shouldn't have slept!

This slow recovery was a low point for our reputation so something had to change. We started experimenting with using rsync to do offsite backups of the mdb files which lead to the development of Yintersync.NET

The things we wanted most from Yintersync.NET were the following:

  • No Database.
  • Fast Transfer.
  • Low bandwidth.
  • Many Systems.

For the last 6 years, we have been using Yintersync.NET to backup our customers Exchange Servers.

Yintersync.NET replicates the Exchange 2007, 2010 and 2013 MDB files. We then robocopy weekly and monthly to 3.5" SATA hard drive in a tray-less hot-plug bay and stored in a Fire Proof Safe. To recover a historical mailbox we simply plug the hard drive into a computer start up Kernel for Exchange Server to pull the necessary PST file out of the MDB.

These archives are useful but we would rather not re-build an Exchange Server. To avoid this we virtualized all our customers Servers on Hyper-V and use Yintersync.NET replicate the VHD's to our Disaster Recovery site.

Thursday, 18 February 2016

Benchmarking Delta Transfers Part 2 : Test Setup & Results

Like our test data, we wanted to make a set-up which is analogous to systems we use in the real world. We also wanted to make it as simple as possible to use and very easy to replicate the same conditions.

We considered the idea of building an entirely virtualised test setup but for simplicity's sake we decided on a physical setup consisting of two off the shelf Dell Vostro 230 systems. A fresh and updated Windows 10 install was put on both systems with the only tweak being separate OS and data partition to avoid fragmentation causing inconsistency on the test space. The system specs were the following.
  • Dell Vostro 230
  • Intel Pentium E6700
  • 4gb DDR3 Memory (2x2gb)
  • 320gb Hard Drive
  • Windows 10 Pro x64
To roughly simulate a network connection, the source PC will have its network interface set to 10 mbit. In an ideal world we would insert a device in the middle (for example pfsense) to artificially add some latency, but for now we are keeping things simple.

The results of our testing are as follows with 1% change to our generated 11 GB data set over a 10 mbit link.

yintersync 2.0.18 (date skip)00:05:08s
yintersync 2.0.18 (hash all)00:09:05s
rsync 3.1.2 cygwin (-vprct --inplace)00:14:42s

Wednesday, 10 February 2016

Benchmarking Delta Transfers Part 1 : Creating the Data

While tweaking Yintersync.NET we have found that bench-marking is tricky as the performance depends entirely on the distribution and size of the changes. What we aim to do here is generate consistent data change which is vaguely analogous to the real world.

In our environment, the rate of change on our servers on a 600 user system is typically around 0.5% per day so we we are going to generate and change data close to those specs.

Our test data set will consist of one big and many small files.
  • 32768 x 32 KB randomly generated files which will have 1% change.
  • single 10 GB file made from merging all the 1 MB files into one file (1% change)
To generate our test data we used Random Data File Creator, and a batch file.

@echo off
set test=d:\test
set tools=c:\tools
mkdir %test%\temp
mkdir %test%\set1\32kb
mkdir %test%\set1\10gb
mkdir %test%\set2\32kb
mkdir %test%\set2\10gb
for /l %%x in (10001,1,42768) do "%tools%\rdfc.exe" "%test%\set1\32kb\32kb%%x.bin" 32 kB overwrite
for /l %%x in (10001,1,20240) do "%tools%\rdfc.exe"  "%test%\temp\1024kb%%x.bin" 1 MB overwrite
copy /b "%test%\temp\1024kb*.bin" "%test%\set1\10gb\10gb-1mb-block.bin" /y
robocopy %test%\set1\32kb %test%\set2\32kb /mir
for /l %%x in (10001,100,42768) do "%tools%\rdfc.exe" "%test%\set2\32kb\32kb%%x.bin" 32 kB overwrite
for /l %%x in (10001,100,20240) do "%tools%\rdfc.exe"  "%test%\temp\1024kb%%x.bin" 1 MB overwrite
copy /b "%test%\temp\1024kb*.bin" "%test%\set2\10gb\10gb-1mb-block.bin" /y
rmdir "%test%\temp" /s /q

This batch file creates two data sets in sub-folders of test path which have a 1% difference.

We now have a data set which we can quickly and consistently modify. In the next post we will setup a system specifically for bench-marking.

Thursday, 4 February 2016

Using Yintersync.NET to provide offsite Hyper-V disaster recovery for nothing (or cheap).

We provide small charities with IT systems and although many have reasonable set-ups, the area that
is often lacking is backup and disaster recovery.

In this case, our customer had a good but modest single server set-up and cloud based backup for most of their data. What was required was an easy and quick way to restore their following server.

Dell Poweredge T110, Windows 2012 Server Standard running 3 Hyper-V VM's.
  • Windows 2012 R2 Domain & File
  • Windows 2012 R2 Remote Desktop 
  • Windows 2008 SQL
Budgets are very limited so we mostly used what they had for the set-up.
  • Dell Vostro 230 (core2duo, 2gb ram, 250gb disk, win7)
  • 2 x 2tb hard drives.
  • Static IP address at backup destination.
  • SSL Certificate.
  • Somewhere to put it, in this case they had a secondary location.
  • Consumer grade broadband.
We built the backup server with the following specifications.
  • Windows 7 on the 250gb hard disk.
  • Software mirror on the 2 x 2gb hard disks.
  • Set-up previous versions at 5pm daily. (see guide)
  • Install Yintersync.NET Server (free for non commercial use).
  • Install the SSL certificate.
  • Set-up appropriate port mapping on their router router.
  • Set-up DNS to point to their static IP address.
  • Set-up Yintersync.NET reporting to send daily emails to the CEO. (see manual).
  • Seed the server with relatively recent copies of the VHD's.
All that was left to do was install the Yintersync.NET agent on their Hyper-V server, link it up and set a 6 pm schedule. (see manual). Easy.

This set-up took us about 2 hours work to put together and now the charity has the following.
  • Automated overnight replica.
  • Easy recovery; set-up the VHD's on a new Windows 2012 system.
  • Roll-back using shadow copies of the VHD files.
  • CEO gets a daily email to inform them their replica is OK.

Saturday, 20 March 2010

Windows Server 2008 Terminal Server, HP Universal Print Driver and Foxit Reader

One of the most frustrating annoyances of Windows terminal server is printing. This is especially the case when there are many different types of printers.

We run a fair few different types of printers and have been struggling with a big print driver set. Printing has worked well enough but shown some signs of instability with occasional print spooler crashing.

Up until recently we used the HP Universal Print Driver for majority of our printers, it a bit bloated but it works well enough. Some printers (other brands and ones incompatible with HP Universal Driver) were supported by their own dedicated drivers.

We then switched from the Adobe Acrobat Reader which performs terribly over thin client to the much better performing Foxit Reader. After doing this we discovered that printing from Foxit to a printer using the HP Universal Driver was very slow. Most of our users didn't notice but one department regularly prints PDF's and it was causing them significant trouble.

Something had to change, either go back to Acrobat viewer and have rubbish PDF viewing performance or find a way to get Foxit to print. Foxit had also appeared to destabilise our printing setup further and was more than enough to push us over the edge and completely rethink our printing solution.

There are various options that would work for us but our plan was to try and find a set of basic PCL print drivers included with Windows that will work with every printer on our system. This was made harder by the fact that nearly all of the more recent HP printers natively supported by Windows Server 2008 use the HP Universal Print Driver, anything after the Laserjet 5 series.

Larger black and white printers were not too much of a problem, for the following printers we used the HP Laserjet 5Si MX Driver which supports these printers fine for us, including duplex where necessary.

HP Laserjet 3005
HP Laserjet 2727 (multifunction)
HP Laserjet 1300
HP Laserjet 4345 (multifunction)
HP Laserjet M4345 (multifunction)
HP Laserjet 1320
HP Laserjet 1200
HP Laserjet P2055
HP Laserjet 2420
HP Laserjet 6P
HP Laserjet M1522
HP Laserjet M9040 (multifunction)
Dell Laserprinter 1720

Some other non HP printers did not work with the 5Si MX driver, these printers are all not duplex so we used the HP Laserjet 4 driver.

Brother HL-2070
Dell Laser Printer 1710
Kyocera FS1920
Samsung ML2251

A few printers did not work correctly with the HP Laserjet 5Si MX driver but worked fine on the Postscript version of it. This does comes at the cost of some print jobs being quite a lot larger and slow over WAN links. The Laserjet 4 driver would probably work on these but that does not support duplex. These printers work fine using the HP Laserjet 5si MX PS Driver.

HP Laserjet 3052 (multifunction)
HP Laserjet 3055 (multifunction)
HP Laserjet 3090 (multifunction)
HP Laserjet 2015

Colour printing was a little more difficult. The Colour Laserjet 5 (The only HP colour printer supported by Windows Server 2008 without the HP Universal Print Driver) does not support duplex. We solved this by switching away from HP print drivers, we used the driver for Dell 3000CN PCL6 which works fine on the following printers.

HP Color Laserjet 3700
HP Color Laserjet CP3525
HP Color Laserjet CP2025
HP Color Laserjet 2605

Some of the printers required some tweaks to the settings to work correctly, scaling documents which no longer fitted in the boundaries of the older printers solved nearly all problems.

Once all this was complete, we went through the registry \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print and removed all unecessary print drivers, monitors and processors and restarted the print spooler. Check out the Hunt for the Bad Printer Driver for greater detail on this.

At last we now only 4 drivers all of which are natively part of Windows Server 2008 which support 21 different printers. A working, reliable print soloution has left us with happy users and even happier admins, for now printing on our network is well under control.