Wednesday, 16 March 2016

Yintersync.NET development update

Things have been a little quiet recently because we are working on some new features.
  • Much faster initial/seed transfer.
  • Exclude files support.
  • Event log in email trimmed to the last 24 hours.
Hopefully we should have a new version within the next 6 weeks with some of these new features.

Tuesday, 8 March 2016

Windows Server 2012 R2 Data Deduplication in Yintersync.NET Hyper-V backup environment.

We wanted to test how well Windows 2012 R2 data deduplication worked with our set of 83 VHD's totaling 7.09TB. We did not want to test on our production disaster recovery setup so we needed a separate test environment.

We installed an extra 4TB Western Digital Red Pro in our existing Server 2012 R2 DR Server and copied on the first 3.5TB of Virtual Machines. We left the system to dedup overnight and copied more virtual machines on. After a few cycles of this our 7.09TB of VHD's fitted onto the 4TB drive using just 3.07TB. of space.

Windows performs deduplication when the system is not busy. The "Chunks" are taken out of the file and and placed in the System Volume Information\Dedup\ChunkStore. Files on the hard drive are now links to the relevant data within the chunkstore. Because of this, you can easily see when the deduplication is complete by the fact that a 200GB file is using "0 bytes" on the disk outside of the ChunkStore.

We next ran our first synchronization over the top of the deduplicated data. The files grew a small amount bigger than "0 bytes". Soon after the sync was complete, the deduplication started again.

Once we starting loading the disk up with lots of simultaneous synchronization, we noticed a significant performance drop when hashing data deduplicated files. With deduplication on a large mechanical disk the capacity to read performance ratio is very poor so the synchronization was unable to complete on our test set of data.

This test failed because we had set unreasonable performance expectations. Even at 130MB/s which is about as fast as the test disk runs in ideal circumstances, hash checking all of this data would take at least 16 hours. We were seeing disk speeds of around 40MB/s, far too slow. In the future we will continue testing Data Deduplication on a RAID 10.

Tuesday, 23 February 2016

Fast, cheap, off site and easy Exchange backup and disaster recovery using Yintersync.NET and Kernel for Exchange

Before we started creating Yintersync.NET one of our customers Exchange Server failed.

The backups were stored in a proprietary database which took around 30 hours to extract, the first attempt of which failed while we were sleeping. The users had to go without an Exchange Server for more than 2 days before we finally got things up and running again. Perhaps we shouldn't have slept!

This slow recovery was a low point for our reputation so something had to change. We started experimenting with using rsync to do offsite backups of the mdb files which lead to the development of Yintersync.NET

The things we wanted most from Yintersync.NET were the following:

  • No Database.
  • Fast Transfer.
  • Low bandwidth.
  • Many Systems.

For the last 6 years, we have been using Yintersync.NET to backup our customers Exchange Servers.

Yintersync.NET replicates the Exchange 2007, 2010 and 2013 MDB files. We then robocopy weekly and monthly to 3.5" SATA hard drive in a tray-less hot-plug bay and stored in a Fire Proof Safe. To recover a historical mailbox we simply plug the hard drive into a computer start up Kernel for Exchange Server to pull the necessary PST file out of the MDB.

These archives are useful but we would rather not re-build an Exchange Server. To avoid this we virtualized all our customers Servers on Hyper-V and use Yintersync.NET replicate the VHD's to our Disaster Recovery site.

Thursday, 18 February 2016

Benchmarking Delta Transfers Part 2 : Test Setup & Results

Like our test data, we wanted to make a set-up which is analogous to systems we use in the real world. We also wanted to make it as simple as possible to use and very easy to replicate the same conditions.

We considered the idea of building an entirely virtualised test setup but for simplicity's sake we decided on a physical setup consisting of two off the shelf Dell Vostro 230 systems. A fresh and updated Windows 10 install was put on both systems with the only tweak being separate OS and data partition to avoid fragmentation causing inconsistency on the test space. The system specs were the following.
  • Dell Vostro 230
  • Intel Pentium E6700
  • 4gb DDR3 Memory (2x2gb)
  • 320gb Hard Drive
  • Windows 10 Pro x64
To roughly simulate a network connection, the source PC will have its network interface set to 10 mbit. In an ideal world we would insert a device in the middle (for example pfsense) to artificially add some latency, but for now we are keeping things simple.

The results of our testing are as follows with 1% change to our generated 11 GB data set over a 10 mbit link.

yintersync 2.0.18 (date skip)00:05:08s
yintersync 2.0.18 (hash all)00:09:05s
rsync 3.1.2 cygwin (-vprct --inplace)00:14:42s

Wednesday, 10 February 2016

Benchmarking Delta Transfers Part 1 : Creating the Data

While tweaking Yintersync.NET we have found that bench-marking is tricky as the performance depends entirely on the distribution and size of the changes. What we aim to do here is generate consistent data change which is vaguely analogous to the real world.

In our environment, the rate of change on our servers on a 600 user system is typically around 0.5% per day so we we are going to generate and change data close to those specs.

Our test data set will consist of one big and many small files.
  • 32768 x 32 KB randomly generated files which will have 1% change.
  • single 10 GB file made from merging all the 1 MB files into one file (1% change)
To generate our test data we used Random Data File Creator, and a batch file.

@echo off
set test=d:\test
set tools=c:\tools
mkdir %test%\temp
mkdir %test%\set1\32kb
mkdir %test%\set1\10gb
mkdir %test%\set2\32kb
mkdir %test%\set2\10gb
for /l %%x in (10001,1,42768) do "%tools%\rdfc.exe" "%test%\set1\32kb\32kb%%x.bin" 32 kB overwrite
for /l %%x in (10001,1,20240) do "%tools%\rdfc.exe"  "%test%\temp\1024kb%%x.bin" 1 MB overwrite
copy /b "%test%\temp\1024kb*.bin" "%test%\set1\10gb\10gb-1mb-block.bin" /y
robocopy %test%\set1\32kb %test%\set2\32kb /mir
for /l %%x in (10001,100,42768) do "%tools%\rdfc.exe" "%test%\set2\32kb\32kb%%x.bin" 32 kB overwrite
for /l %%x in (10001,100,20240) do "%tools%\rdfc.exe"  "%test%\temp\1024kb%%x.bin" 1 MB overwrite
copy /b "%test%\temp\1024kb*.bin" "%test%\set2\10gb\10gb-1mb-block.bin" /y
rmdir "%test%\temp" /s /q

This batch file creates two data sets in sub-folders of test path which have a 1% difference.

We now have a data set which we can quickly and consistently modify. In the next post we will setup a system specifically for bench-marking.

Thursday, 4 February 2016

Using Yintersync.NET to provide offsite Hyper-V disaster recovery for nothing (or cheap).

We provide small charities with IT systems and although many have reasonable set-ups, the area that
is often lacking is backup and disaster recovery.

In this case, our customer had a good but modest single server set-up and cloud based backup for most of their data. What was required was an easy and quick way to restore their following server.

Dell Poweredge T110, Windows 2012 Server Standard running 3 Hyper-V VM's.
  • Windows 2012 R2 Domain & File
  • Windows 2012 R2 Remote Desktop 
  • Windows 2008 SQL
Budgets are very limited so we mostly used what they had for the set-up.
  • Dell Vostro 230 (core2duo, 2gb ram, 250gb disk, win7)
  • 2 x 2tb hard drives.
  • Static IP address at backup destination.
  • SSL Certificate.
  • Somewhere to put it, in this case they had a secondary location.
  • Consumer grade broadband.
We built the backup server with the following specifications.
  • Windows 7 on the 250gb hard disk.
  • Software mirror on the 2 x 2gb hard disks.
  • Set-up previous versions at 5pm daily. (see guide)
  • Install Yintersync.NET Server (free for non commercial use).
  • Install the SSL certificate.
  • Set-up appropriate port mapping on their router router.
  • Set-up DNS to point to their static IP address.
  • Set-up Yintersync.NET reporting to send daily emails to the CEO. (see manual).
  • Seed the server with relatively recent copies of the VHD's.
All that was left to do was install the Yintersync.NET agent on their Hyper-V server, link it up and set a 6 pm schedule. (see manual). Easy.

This set-up took us about 2 hours work to put together and now the charity has the following.
  • Automated overnight replica.
  • Easy recovery; set-up the VHD's on a new Windows 2012 system.
  • Roll-back using shadow copies of the VHD files.
  • CEO gets a daily email to inform them their replica is OK.