Seven Biggest Challenges of File Migration, And the Optimal Tool to Surmount Them.


File migration issues date back to early computing days of moving databetweenpunch cards andgiantstoragedisc technologies. In more recent years, understandably, the issues have been more notably associated with NAS-to-NAS, server-to-NASand server-to-serverfile-based migration processes.


I have worked with quite a few tools in the last 20 years, notably ROBOCOPY, EMCOPY, RSYNC, VNX Replicator, Isi_vol_copy, etcetera. While ROBOCOPY, EMCOPY and RSYNC were quite handy in my beginning years with platform migration, VNX Replicator and Isi_vol copy came handy with later technologies. Below is a peep into the functional summary of above mentioned tools. ROBOCOPY was bundled with Windows NT 4.0 Resource Kit. You will recall the prevalence of this Microsoft network operating system in the late 1990’s. EMCOPY is more of a specialised tool for file systems that were based on a Celerra; however, the other tools in the suite comprised of EMCopy. Some other tools which were bundled with EMCOPY include Shared up and LGDUP. These tools could be used vairably to complement EMCOPY or other independent tools such as ROBOCOPY. RSYNC ison the Open System platforms. Essentially, it isa UNIX tool, majorly used for the migration of NFS.

The Challenges and the Solutions

Below are thesome of the prevailingbiggest challenges most frequently associated with file migration: Accelerated use of bandwidth, Need for Detailed Reporting, Need for job scheduling, Ability to estimate switchover time, Need for creation of migration jobs Incremental copies, Mix of SMB and NFX clients , We will now look at the toolsthat have identified, tried and tested; to optimally assist in surmounting the challenges itemized above.

Accelerated Use of Bandwidth

Migrations can be bandwidth intensive. As such during migration there may be a need to adjust the allocation of bandwidth demanded by the migration process. The foregoing may then affect the bandwidth available for routine daily work and the concurrent migration sessions; and the amount of time required to complete the entire process. The issue above is addressed by DobiMiner. This file migration tool can dynamically schedule bandwidth in such a way as to take care of operation hours per file server in question.

The effect of the above is that during normal working hours (when bandwidth demand by ordinary users is intensive) the migration would be running with reduced and capped bandwidth. However, this would automatically change after business hours, when general bandwidth demand drops significantly, as bandwidth allocation to the migration process would automatically increase to enable the acceleration of the migration of the files.

Put differently, the above affords the opportunity to have elastic usage of bandwidth at the different times of the day, and have the flexibility to change schedules based on the type and size of file server encountered by the migration processes.Additionally, the procedure has the capability of file server scans for data refresh.

Detailed Reporting

  • As in most IT jobs, there is often need to have reports showing successes, failures and the details underliningthe successes and failures of the migration job. And here is another salient capability of DobiMiner.

  • It can generate reports on demand, and on schedule. This is a very valuable attribute which answers to the request to see a summarising daily status report on all migration jobs.

  • DobiMiner stands heads and shoulders above other available tools in this respect, some of which would require some adjustments to produce such reports.

  • In addition to the source and target destination, the reports can granulate to the number of files, directories, symbolic links and errors. Furthermore, in the event of an error, the reports provide the added capability to drill further down to analyse the cause.

  • Additionally, DobiMiner allows for the scheduling of these Migration reports to be emailed daily, weekly, or monthly.

Estimating Switchover Time

Estimating Switchover Time is a manual process, which includes the review of the logs from each incremental run, assessing the time to perform the incremental copy. To enable the above process, DobiMiner has the capability to create migration gaps to address these reviews. Also, DobiMiner has the capability to regulate the final migration into specified time window, which can be verified using a dry run simulation.

Creation of Migration Jobs

After job conception and arranging (which could be automated using scripts), the next task is to assign them to the servers that should accomplish the tasks, and then schedule them. DobiMiner provides the functionality to import the files in bulk; and their migration pairs from a template.

Incremental Copies

Like many other migration tools, DobiMiner performs incremental copies. However, in addition, DobiMiner performs a scan of the directory (faster than the other tools) on a scheduled basis; and stores information in its filesystem. This way, DobiMiner performs migration faster than the other tools.

Mix of SMB and NFS Clients

DobiMine reliminates the need for multiple tools by achieving migration of both NFS and SMB files in a single ‘glass-pane’; two functions that alternatively would be performed using two tools like Robocopy and Rsynch. It also supports mixed-mode protocols.