ESXi Migration: Or How I Learned to Stop Worrying and Check My NFS Config

TL;DR: I spent days thinking my HDD was too slow for ESXi VM migration. Turns out I just forgot to reload my NFS server config. Don't be like me. Also, benchmarks inside.

Units: all sizes/speeds use binary units (powers of two). In this doc: 1 MB = 2^20 bytes, 1 GB = 2^30 bytes. (No marketing units.)

-> Benchmark charts & full data matrix

-> 💰 Advanced deep dive: full matrix, tmpfs experiments, vmkfstools ceiling, sync anomaly

Why This Guide Exists (And Why You Can't Just dd Your Way Out)

"Can't I just mount the VMFS volume on Linux and copy the files?"

No. Here's why you're stuck with this approach:

The Fundamental Problems

1. No supported way to mount VMFS6 on Linux

   • VMware doesn't ship a Linux VMFS6 driver, and there's no in-kernel support
   • Third-party projects exist (e.g., vmfs6-tools / vmfs-fuse), but they're typically read-only and fragile -- fine for last-ditch recovery, not a predictable migration runbook

2. ESXi won't use "normal Linux filesystems" as datastores

   • For datastores: VMFS (block) or NFS (file)
   • Removable media is a different story (VFAT/FAT32; sometimes ext3 via CLI), but it's not a great place to park large VMs

3. Cloning the disk doesn't help

   • dd if=/dev/esxi-disk of=/dev/new-disk copies the VMFS filesystem
   • Linux still can't use it
   • You're back to square one

4. Passthrough doesn't magically solve it

   • RDM / passthrough gives a Linux VM the block device
   • You're still stuck at "how do I read VMFS6?"

The Practical Approach (If You Don't Have vCenter)

Mount an NFS datastore on ESXi -> Use vmkfstools to copy VMDKs off VMFS

This works because:

• ESXi can read its own VMFS (obviously)
• ESXi can write to NFS shares
• vmkfstools runs inside ESXi and understands VMDK formats
• Linux can mount NFS and access the copied files
• You can then convert VMDK -> qcow2/raw on Linux with qemu-img

Alternative scenarios:

• Fresh ESXi install from USB: If you only have the datastore disk, boot ESXi from USB, add NFS storage, copy VMs out (no license needed for this)
• USB disk as scratch target: Removable-media support on ESXi is limited (VFAT/FAT32; sometimes ext3 via CLI). It can work in a pinch, but it's awkward for large VM exports and not consistently supported -- NFS is usually simpler and faster.
• OVF/OVA export tools: exist, but for bulk migration of hundreds of VMDKs they are not a sane automation surface. Stick to vmkfstools over an NFS datastore.
• vCenter with shared storage: Use Storage vMotion (requires licenses you probably don't have)
• VMware Converter: Requires Windows, vCenter, licenses, and still uses network transfer
• Starwind V2V: Works but still requires mounting the VMFS somehow (catch-22*)

* Catch-22: A paradoxical situation where you need X to get Y, but need Y to get X. You need to mount VMFS to use V2V tools, but you need V2V tools to access VMFS without ESXi.

Why This Guide Matters

In practice, it's vmkfstools over NFS. The question is: do you want it to take 3 minutes or 3 hours per VM?

That's what this guide is about.

The Problem That Wasn't

I needed to migrate VMs off an ESXi host. Standard approach: mount an NFS datastore as the target, then use vmkfstools to copy VMDKs out of VMFS and onto the NFS share.

Expected speed: Maybe 100 MB/s?
Actual speed: 3-5 MB/s
My conclusion: "The HDD is dying from all these random reads"

So I did what any reasonable person would do: ordered a 4TB SSD for $350, cloned the entire VMFS datastore to it, and prepared to write a guide about how SSDs are essential for ESXi migration.

Spoiler: I'm an idiot.

The "Aha" Moment (That Took Too Long)

After getting the SSD and seeing great speeds (98 MB/s), I decided to benchmark the HDD one more time for comparison.

Result: 82 MB/s.

Wait, what?

Same HDD. Same VM. Same everything. Except now it was only 20% slower than the SSD, not 95% slower.

What changed?

I'd edited /etc/exports to use async mode for better performance... but I never ran exportfs -ra to reload the config. The NFS server was still running with sync mode from the previous config.

The Real Culprit: NFS Sync Mode

Here's what sync mode does:

• Every write must hit physical disk before acknowledging to the client
• No write caching, no buffering
• Safe for data integrity
• Absolutely murders performance

Here's what async mode does:

• Writes are buffered in RAM and acknowledged immediately
• NFS can batch writes efficiently
• Less safe if you lose power mid-transfer
• 13-24x faster for bulk data transfer -- measured, not estimated

The performance numbers, benchmarked across real hardware:

The cruelest entry: SSD->NVMe over 25G with sync enabled. You bought all the right hardware, built the perfect stack -- and a single config option turns 290 MB/s into 12 MB/s. The sync penalty actually grows as your hardware gets faster, because there's more potential being wasted.

The lesson: Forgetting to run exportfs -ra cost me 13-24x in performance, not the HDD.

What I Actually Learned (With Benchmarks)

-> Full data matrix and isolated variable analysis

Test Setup

The full matrix covers all combinations of source x destination x NIC speed x MTU -- 16 runs total, same VM each time -- 32 GB provisioned, ~19 GB allocated (~41% sparse). Folder naming tells you everything: esxi_{src}_nfs_{dst}_{nic}_mtu{mtu}.

• Source: WD Gold WD4003FRYZ (HDD) · Samsung 870 QVO 4TB (SSD)
• Destination: WD Gold RAID array (HDD) · KIOXIA Exceria Pro (NVMe)
• NIC: onboard 1G · Intel XXV710-DA2 25G, direct DAC cable
• MTU: 1500 (standard) · 9000 (jumbo), both tested at both speeds
• VM: k8s_master.vmdk -- 32 GB provisioned, ~19 GB on-disk (du), ~41% sparse (≈1.68x thinness)
  • Note: throughput and "MB/s" figures are computed from the allocated bytes (~19 GB), because vmkfstools -d thin copies allocated blocks. If your disks are thick-provisioned, expect times to scale roughly with the provisioned size.
  • How that was measured on Linux:
    
    
    

    du -h k8s_master-flat.vmdk  # ~19G on disk
    
    du -h --apparent-size k8s_master-flat.vmdk  # ~32G provisioned
    
    
    

    
    
    

Full Results Matrix

What The Data Proves

MTU 9000 vs 1500 -- the free lunch:

Consistent +3-4 MB/s absolute gain across every single combination. The percentage looks better on slower setups (~4% on 1G vs ~1.5% on 25G+NVMe) -- not because the lemon got juicier, just because the glass was smaller. Set it. It is free.

NVMe destination -- context is everything:

On 1G: NVMe vs HDD = +0.4 MB/s. That is not a rounding error, that is a wall called 1G. The ceiling does not care what is behind it. On 25G + SSD source: +130 MB/s, +84%. Completely different story. Do not upgrade the destination in isolation -- upgrade the full stack.

25G network -- the bigger the pipe, the more storage matters:

With HDD source, 25G buys ~30%. With SSD+NVMe, 25G is a 3x multiplier. The network upgrade does not help much with slow storage -- there is nothing to carry. But once the storage can actually feed it, the pipe pays off.

SSD source -- latency matters more than you think:

On 1G: SSD is 1.2x faster than HDD. Network is the wall, both drives can feed it easily. On 25G+NVMe: SSD is 2.3x faster. The SSD's 1.5ms read latency vs HDD's 9ms finally has room to matter -- vmkfstools pipelines better, the network stops being the lid, and everything accelerates together.

Upgrades multiply, not add:

Stack all four: 82 -> 290 MB/s. 3.5x. These are 16 data points, not theory.

Source IOPS (ESXi esxtop SCSI counters)

You'll notice source read MB/s is consistently higher than the wire throughput MB/s in the table above. This is expected: vmkfstools -d thin scans the full provisioned range of the VMDK to find allocated blocks, but only transmits the allocated ones over the wire. For a ~41% sparse disk, ESXi reads ~32 GiB worth of blocks from the source device while only ~19 GiB travels over the network. Source IOPS reflect the scan; wire throughput reflects what was actually sent.

Destination IOPS (Linux iostat)

NVMe at 0.2% utilisation doing 4,700+ IOPS. It could run this workload 500 times over. It is not the bottleneck. It will never be the bottleneck.

Why The HDD Didn't Suck

My WD Gold is an enterprise drive designed for this kind of workload:

• 7200 RPM (not slow consumer 5400 RPM)
• Large cache (256 MB)
• Firmware optimized for random I/O
• Rated for 24/7 datacenter use

But more importantly, vmkfstools' read pattern isn't actually tiny random 4K reads:

• Average read size: ~64 KB chunks
• Sequential reads within each VMDK extent
• Read-ahead caching helps
• 2,000+ IOPS is achievable on enterprise HDDs

Consumer desktop drives (WD Blue, Seagate Barracuda) would perform worse - probably 40-60 MB/s instead of 82 MB/s. But still infinitely better than 3-5 MB/s with sync mode.

The SSD Staging Approach (Still Valid, But Context Matters)

Even though the HDD performed OK, the SSD staging approach still has merit - but the reason depends on your network:

Why SSD staging makes sense on 1G:

• 19% faster (98 vs 82 MB/s) - modest but real
• Eliminates variables (consistent performance regardless of HDD quality)
• Consumer HDDs would show a bigger gap

Why SSD staging REALLY makes sense on 25G:

• 129% faster when paired with NVMe destination (286 vs 125 MB/s)
• Source latency (1.5ms vs 9ms) becomes the critical factor
• HDD's 9ms latency limits pipelining the 25G link can handle
• The advantage scales dramatically with network speed

When you DON'T need SSD staging:

• You have 1G network and enterprise HDDs (82 MB/s is fine for most migrations)
• Your migration window isn't tight
• You remembered to enable NFS async (check it twice!)

The upgrade decision matrix

The Step-by-Step Guide (With All The Checks)

Step 0: Don't Be Like Me

Before you do ANYTHING else, let's make sure NFS is configured correctly.

On your Linux NFS server:

# Edit /etc/exports - USE ASYNC NOT SYNC
cat > /etc/exports << 'EOF'
/mnt/raid/esxi-import 192.168.0.0/24(rw,async,no_root_squash,no_subtree_check)
EOF

# CRITICAL: Actually reload the config
exportfs -ra

# Verify it actually applied
exportfs -v | grep async
# Should show: /mnt/raid/esxi-import 192.168.0.0/24(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,async,...)

# If you don't see 'async' in the output, you messed up - fix it before continuing

⚠️ IMPORTANT: NFS Async Safety Warning

The async mode buffers writes in RAM before writing to disk. This means:

• If you lose power during transfer: Buffered data in RAM is lost
• If the NFS server crashes: Last few seconds of writes may be lost
• If network drops: Transfer stops but no data corruption (just incomplete)

How to mitigate the risk:

1. Use a UPS (Uninterruptible Power Supply) on the NFS server - battery backup that keeps server running during power outages
2. Don't run other services on the NFS server during migration
3. Verify your copies with md5sum after transfer completes
4. Keep source VMs intact until you've verified the copies work

Why async is still worth it:

• 13-24x faster in measured runs (and yes, if you're stuck at ~3-5 MB/s, it'll feel like 20x+)
• Modern Linux buffers are flushed frequently (every 5-30 seconds)
• Risk window is small (seconds, not minutes)
• You're keeping the source anyway until migration is verified

⚠️ WARNING END

Seriously, check this three times. I'm not kidding. Print it out. Tattoo it on your arm. Don't skip this.

Step 1: Verify Your Source Drive Type

Figure out what you're working with:

# On ESXi
esxcli storage core device list | grep -A 20 "Display Name.*Local"

# Look for the model number, something like:
# - WD4003FRYZ = WD Gold (enterprise, good)
# - WD40EFRX = WD Red (NAS-grade, OK)  
# - WD10EZEX = WD Blue (desktop, might struggle)
# - ST4000DM004 = Seagate Barracuda (consumer, slower)

If you have an enterprise drive AND proper NFS async, you can skip the SSD staging. Just migrate directly.

If you have a consumer drive, SSD staging will help more (might be 2-3x faster instead of just 20%).

Step 2: (Optional) SSD Staging Setup

If you decide SSD staging is worth it for your situation:

# Verify SSD is big enough BEFORE cloning
df -h /vmfs/volumes/your-source-datastore
blockdev --getsize64 /dev/your-ssd-device

# If SSD < datastore size, STOP and buy a bigger SSD
# Don't ask me how I know this

Clone VMFS to SSD:

# On Linux server
dd if=/dev/source-disk of=/dev/ssd bs=1M status=progress
# Or use ddrescue for better error handling

Resign the cloned VMFS on ESXi:

# Check for snapshots requiring signature
esxcli storage vmfs snapshot list

# Should show:
# Volume Name: datastore-name  
# Can resignature: true

# Mount it with new signature
esxcli storage vmfs snapshot mount -l datastore-name

# Verify it mounted
esxcli storage vmfs extent list

Common issue: "duplicate extents found" error means you have the same VMFS UUID on multiple disks.

# Find which disks have VMFS partitions
esxcli storage vmfs extent list

# Wipe the partition table on the disk you DON'T want
partedUtil mklabel /vmfs/devices/disks/t10.WRONG_DISK_ID gpt

# Rescan storage
esxcli storage core adapter rescan --all

# Try resignature again

Step 3: Mount NFS on ESXi

# On ESXi
esxcli storage nfs add -H 192.168.0.105 -s /mnt/raid/esxi-import -v migration_target

# Verify mount
esxcli storage nfs list
# Should show: Mounted: true

# Common mistakes:
# - Typo in path (esxi-import vs exsi-import) -> mount denied
# - Firewall blocking port 2049 -> timeout
# - Wrong subnet in /etc/exports -> permission denied

Test from another Linux machine first if you're paranoid (you should be):

showmount -e 192.168.0.105
mount -t nfs 192.168.0.105:/mnt/raid/esxi-import /mnt/test

Step 4: Migrate VMs

Consistency rule: don't copy live disks unless you're deliberately taking that risk. Power the VM off, or copy from a snapshot chain (quiesced if you can). vmkfstools -i is a copier, not a consistency layer.

# For each VM disk
time vmkfstools -i /vmfs/volumes/source/vm/disk.vmdk \
  -d thin \
  /vmfs/volumes/migration_target/vm/disk.vmdk

Why not cp / scp / "download from the datastore browser"?

Those paths often read the logical size of a thin VMDK and write it back out as a fully-allocated file -- i.e., "32 GB provisioned" turns into "32 GB transmitted and stored", even if only ~19 GB was actually allocated. If you care about preserving sparseness (and keeping the migration fast), stick to vmkfstools -i for the copy step.

Expected performance (assuming NFS async is enabled):

If you're seeing 3-5 MB/s, STOP. You forgot to enable async or forgot to reload the config. Go back to Step 0.

Step 5: Validate Copies

Hash while both datastores are still mounted on ESXi, saving the checksum file to the NFS target alongside the copy. Then you can power down the source and verify cold on the Linux side — no cache ambiguity, no "same mount" shortcut.

# On ESXi — hash the source flat file, save checksum to NFS target
md5sum /vmfs/volumes/source-datastore/vm/disk-flat.vmdk > /vmfs/volumes/migration_target/vm/disk-flat.vmdk.md5

# Power down / unmount source when done with all VMs

# On Linux — hash the destination flat file and compare against the saved checksum
md5sum /mnt/raid/esxi-import/vm/disk-flat.vmdk
cat /mnt/raid/esxi-import/vm/disk-flat.vmdk.md5
# Both lines should show the same hash

The .md5 file lives on the NFS share next to the copy, so it travels with the data and doubles as a permanent audit trail.

For large disks, spot-check before committing to a full hash:

# First 100MB (on ESXi, against source)
head -c 100M /vmfs/volumes/source-datastore/vm/disk-flat.vmdk | md5sum

# Same on Linux, against destination
head -c 100M /mnt/raid/esxi-import/vm/disk-flat.vmdk | md5sum

Troubleshooting: The Checklist

If migration is slow (< 20 MB/s):

1. ☐ Did you edit /etc/exports to use async?
2. ☐ Did you run exportfs -ra after editing?
3. ☐ Did you verify with exportfs -v | grep async?
4. ☐ Did you restart the NFS client mount on ESXi after changing server config?
5. ☐ Is your network actually 1G or did someone plug a 100M cable in?
6. ☐ Are you testing with a tiny VM that fits in cache vs a large one?

If you get "duplicate extents" error:

# You have the same VMFS UUID on multiple disks
# Find the duplicate:
esxcli storage vmfs extent list

# Wipe the wrong one:
partedUtil mklabel /vmfs/devices/disks/t10.WRONG_DISK_ID gpt

# Rescan and try again
esxcli storage core adapter rescan --all

If NFS mount fails:

# Check exports are active
exportfs -v  

# Check firewall isn't blocking
iptables -L -n | grep 2049

# Test from another Linux machine
showmount -e 192.168.0.105

The Economics: Is SSD Staging Worth It?

My situation:

• Cost: $350 for 4TB Samsung 870 QVO
• Time saved: 20% (4 min -> 3 min per VM)
• Actual benefit: Eliminated risk of consumer HDD being slow
• Bonus: I now own a 4TB SSD

When it's worth it:

• Large migration (100+ VMs)
• Tight migration window
• Unknown/consumer source drives
• You want guaranteed performance
• You can repurpose the SSD afterward

When it's not worth it:

• Small migration (< 10 VMs)
• You have enterprise HDDs
• Tight budget
• Migration can run overnight

The real cost comparison:

What About Commercial Migration Tools?

Veeam, StarWind, VMware Converter - they all do the same thing under the hood:

• Read blocks from source VMFS
• Write blocks to destination
• Can't fix physics (NFS sync, slow HDDs, slow network)

They might add:

• Pretty progress bars
• Incremental/differential copies
• Deduplication
• Parallel VM migration
• Your company's money ($$$)

But they won't magically make sync mode fast or turn your WD Blue into a WD Gold.

For a one-time migration, spending $350 on an SSD beats spending $5,000 on Veeam licenses.

The Network Factor

Both my tests hit network limits:

With a 10G network:

• Expected: 200-300 MB/s (destination RAID becomes bottleneck)
• SSD would help more (can deliver 500+ MB/s)

With 25G + NVMe both ends:

• Proven: 286 MB/s (SSD source -> NVMe dest, 25G MTU9000)
• Full data: benchmark results

For 1G networks, the SSD vs HDD difference is muted by network limits.

Lessons Learned (The Hard Way)

1. Configuration mistakes cost more than hardware

• Forgetting exportfs -ra: 13-24x penalty in measured runs (and up to ~27x if you're in the 3-5 MB/s pit)
• Buying an SSD to "fix" it: $350
• Feeling stupid when you realize: Priceless

2. Check your assumptions with benchmarks

• "The HDD is too slow" -> Actually fine with proper config
• "I need an SSD" -> Only 20% faster in practice
• "This will take 2 weeks" -> Takes 4 hours with correct setup

3. Enterprise hardware has a reason to exist

• WD Gold handled 2,174 IOPS just fine
• Consumer drives would struggle more
• But config matters more than hardware

4. Document your changes

When you edit a config file at 2 AM:

• ☐ Make the change
• ☐ Reload the service
• ☐ Verify it applied
• ☐ Test it works
• ☐ Write down what you did

Skip any step and you'll be buying unnecessary SSDs.

5. The checklist is your friend

Before declaring "the hardware is slow":

1. Is the config actually loaded?
2. Is the service actually restarted?
3. Did I typo the path?
4. Is the firewall blocking it?
5. Did I test from another machine?

If answer to any is "uh... maybe?", go check.

The Updated Guide Philosophy

Original plan: "Buy SSD, use SSD, go fast"

Reality: "Enable NFS async, double-check you enabled it, triple-check it actually loaded, THEN decide if you need an SSD"

The SSD staging approach is still valid - it eliminates variables and guarantees performance. But it's optimization, not a requirement.

The requirement is proper NFS configuration.

Conclusion

If you take away one thing from this guide:

NFS sync mode will ruin your day, your week, and your migration project.

Enable async. Reload the config. Verify it loaded. Test it works.

Then worry about whether you need an SSD.

And if you do buy an SSD for staging, at least you'll actually get the performance you paid for, unlike me who got the same speed from the HDD after fixing the config.

Appendix: The Actual Commands (Copy-Paste Ready)

NFS Server Setup

# /etc/exports
/mnt/raid/esxi-import 192.168.0.0/24(rw,async,no_root_squash,no_subtree_check)

# Apply changes (DON'T FORGET THIS)
exportfs -ra

# Verify async is enabled (DO THIS TOO)
exportfs -v | grep async

# Start NFS server
systemctl enable --now nfs-server

ESXi NFS Mount

# Mount NFS
esxcli storage nfs add -H 192.168.0.105 -s /mnt/raid/esxi-import -v migration_target

# Verify
esxcli storage nfs list

Migrate VM Disk

# Thin provisioning (copies only used blocks)
vmkfstools -i /vmfs/volumes/source/vm/disk.vmdk \
  -d thin \
  /vmfs/volumes/migration_target/vm/disk.vmdk

# Thick eager zeroed (faster but uses full space)
vmkfstools -i /vmfs/volumes/source/vm/disk.vmdk \
  -d eagerzeroedthick \
  /vmfs/volumes/migration_target/vm/disk.vmdk

Validate

# Hash the flat files
md5sum /vmfs/volumes/source/vm/disk-flat.vmdk
md5sum /vmfs/volumes/migration_target/vm/disk-flat.vmdk

Licence: CC BY 4.0 — free to use with attribution.

Last updated: February 2026 (and won't be updated again)

Author: A sysadmin who spent $350 on an SSD to fix a configuration problem, so you don't have to

Disclaimer: This worked for me. Your mileage may vary. Back up your data. Don't blame me if something breaks. Seriously, back up your data first.

P.S. For the Homelab Folks

Yes, people run desktop HDDs in ESXi servers. I've seen:

• "Temporary" repurposed gaming drives (permanent)
• Random WD Blues in whitebox builds
• Seagate Barracudas "because I had them"
• "It works in my PC" logic

If you're running consumer drives:

• NFS async is mandatory (not optional)
• SSD staging will help more (maybe 2-3x faster)
• Consider upgrading to WD Red/Red Pro for ~$50 more
• Or just buy a used enterprise drive on eBay for $30

But most importantly: Check your NFS config three times.

You're welcome.