Overview
This comprehensive guide covers migrating a complete website using three essential tools:
- rsync for file transfers
- MySQL dump for database migration
- imapsync for email migration
Essential Documentation Links
- rsync Official Manual
- MySQL 8.0 mysqldump Reference
- imapsync Official Website
- imapsync GitHub Repository
Pre-Migration Planning
1. Infrastructure Assessment
- Document current server specifications and configurations
- Identify all websites, databases, and email accounts
- Plan DNS change strategy and TTL reduction
- Set up SSH key authentication between servers
- Verify sufficient disk space on destination server
2. Downtime Planning
- Schedule migration during low-traffic periods
- Coordinate with users about potential email interruptions
- Plan for incremental transfers to minimize downtime
- Prepare rollback procedures
3. Security Setup
|
1 2 3 4 5 6 7 8 9 10 |
# Generate SSH key pair (if not already exists) ssh-keygen -t rsa -b 4096 -C "migration@yourserver.com" # Copy public key to destination server ssh-copy-id user@destination-server.com # Test SSH connection ssh user@destination-server.com |
Reference Links:
File Migration with rsync
Basic rsync Configuration
Standard Website Files Transfer
|
1 2 3 4 5 6 7 8 9 10 |
# Basic rsync command for website files rsync -avh --progress /var/www/html/ user@destination:/var/www/html/ # With SSH and compression rsync -avhz --progress -e ssh /var/www/html/ user@destination:/var/www/html/ # Dry run first (recommended) rsync -avh --dry-run --progress /var/www/html/ user@destination:/var/www/html/ |
Advanced rsync Options
|
1 2 3 4 5 6 7 8 |
# Complete migration command with excludes rsync -avhz --progress --delete --exclude='*.log' \ --exclude='cache/*' --exclude='tmp/*' \ --exclude='.git/*' \ -e "ssh -o StrictHostKeyChecking=no" \ /var/www/html/ user@destination:/var/www/html/ |
|
1 2 3 4 5 6 |
# What I mostly use! rsync -chavzP -e "ssh -p 222" ~/public_html user@server:public_html/ |
Key rsync Flags Explained
Flag | Purpose |
|---|---|
-a | Archive mode (preserves permissions, timestamps, symbolic links) |
-v | Verbose output |
-h | Human-readable file sizes |
-z | Compress data during transfer |
-c | Skip based on checksum, not mod-time & size |
-h | Output numbers in a human-readable format |
-e | specify the remote shell to use |
--progress, -P | Show transfer progress |
--delete | Delete files on destination that don’t exist on source |
--exclude | Exclude specific files/directories |
--dry-run | Test run without making changes |
Performance Optimization
For Servers with Many Small Files
|
1 2 3 4 5 6 7 8 9 |
# Clean up before migration find /var/www/html -name "*.tmp" -delete find /var/www/html -name "session_*" -delete find /var/www/html/cache -type f -delete # Use parallel processing for small files rsync -avhz --progress --inplace --whole-file /source/ user@dest:/target/ |
For Large Files
|
1 2 3 4 5 |
# Use partial transfers for large files rsync -avhz --progress --partial --partial-dir=.rsync-partial \ /source/ user@dest:/target/ |
Two-Phase Migration Strategy
|
1 2 3 4 5 6 7 8 9 |
# Phase 1: Initial bulk transfer (while site is live) rsync -avhz --progress /var/www/html/ user@destination:/var/www/html/ # Phase 2: Final sync (during maintenance window) # Stop web services first systemctl stop apache2 # or nginx rsync -avhz --progress --delete /var/www/html/ user@destination:/var/www/html/ |
Additional Resources:
Database Migration with MySQL dump
InnoDB Databases (Recommended)
|
1 2 3 4 5 6 7 8 9 10 11 |
# Single database dump with consistency mysqldump --single-transaction --skip-lock-tables --quick \ --routines --triggers --events \ -u username -p database_name > database_dump.sql # All databases mysqldump --single-transaction --skip-lock-tables --quick \ --routines --triggers --events --all-databases \ -u root -p > all_databases.sql |
MyISAM Databases (Legacy)
|
1 2 3 4 5 6 |
# MyISAM requires table locking mysqldump --lock-tables --quick \ --routines --triggers --events \ -u username -p database_name > database_dump.sql |
Large Database Optimization
|
1 2 3 4 5 6 7 8 9 10 |
# For databases > 10GB, use compression and streaming mysqldump --single-transaction --skip-lock-tables --quick \ -u username -p database_name | gzip > database_dump.sql.gz # Direct transfer to remote server mysqldump --single-transaction --skip-lock-tables --quick \ -u username -p database_name | \ ssh user@destination "mysql -u username -p target_database" |
MySQL Resources:
Critical mysqldump Options
Option | Purpose |
|---|---|
--single-transaction | Creates consistent snapshot for InnoDB |
--skip-lock-tables | Prevents table locking |
--quick | Retrieves rows one at a time (memory efficient) |
--routines | Includes stored procedures and functions |
--triggers | Includes triggers |
--events | Includes scheduled events |
--master-data | Includes binary log position (for replication) |
Database Restoration
|
1 2 3 4 5 6 7 8 9 10 |
# Basic restoration mysql -u username -p target_database < database_dump.sql # With progress monitoring using pv pv database_dump.sql | mysql -u username -p target_database # Compressed file restoration gunzip < database_dump.sql.gz | mysql -u username -p target_database |
Large Database Restoration Optimization
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# Optimize MySQL settings for restoration mysql -u root -p -e " SET foreign_key_checks = 0; SET unique_checks = 0; SET autocommit = 0; SOURCE database_dump.sql; COMMIT; SET foreign_key_checks = 1; SET unique_checks = 1; SET autocommit = 1;" |
Email Migration with imapsync
Basic imapsync Setup
|
1 2 3 4 5 6 7 8 9 10 11 |
# Install imapsync (Ubuntu/Debian) apt-get update apt-get install imapsync # Basic migration command imapsync --host1 source-mail.example.com --port1 993 --ssl1 \ --user1 user@example.com --password1 'sourcepass' \ --host2 dest-mail.example.com --port2 993 --ssl2 \ --user2 user@example.com --password2 'destpass' |
imapsync Resources:
Secure Configuration with Password Files
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Create password files (more secure) echo 'sourcepassword' > /tmp/pass1 echo 'destpassword' > /tmp/pass2 chmod 600 /tmp/pass1 /tmp/pass2 # Migration with password files imapsync --host1 source-mail.example.com --port1 993 --ssl1 \ --user1 user@example.com --passfile1 /tmp/pass1 \ --host2 dest-mail.example.com --port2 993 --ssl2 \ --user2 user@example.com --passfile2 /tmp/pass2 \ --delete2 |
Advanced imapsync Options
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Complete migration with advanced options imapsync --host1 source.example.com --ssl1 --port1 993 \ --user1 user@example.com --passfile1 /tmp/pass1 \ --host2 dest.example.com --ssl2 --port2 993 \ --user2 user@example.com --passfile2 /tmp/pass2 \ --exclude 'Spam|Junk|Trash' \ --syncinternaldates \ --idatefromheader \ --delete2 \ --expunge1 \ --dry |
Key imapsync Options
Option | Purpose |
|---|---|
--ssl1/--ssl2 or --tls1 --tls2 | Use SSL / TLS connections |
--delete2 | Delete messages on destination not on source |
--syncinternaldates | Preserve internal message dates |
--idatefromheader | Use date from message headers |
--exclude | Skip specific folders |
--dry | Perform dry run without changes |
--expunge1 | Permanently delete messages marked for deletion |
Bulk Email Migration Script
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
#!/bin/bash # migrate_all_emails.sh ACCOUNTS_FILE="email_accounts.txt" SOURCE_HOST="old-mail.example.com" DEST_HOST="new-mail.example.com" while IFS=',' read -r email password; do echo "Migrating $email..." echo "$password" > /tmp/pass_temp chmod 600 /tmp/pass_temp imapsync --host1 "$SOURCE_HOST" --ssl1 --port1 993 \ --user1 "$email" --passfile1 /tmp/pass_temp \ --host2 "$DEST_HOST" --ssl2 --port2 993 \ --user2 "$email" --passfile2 /tmp/pass_temp \ --syncinternaldates \ --delete2 rm /tmp/pass_temp echo "Completed $email" done < "$ACCOUNTS_FILE" |
Migration Workflow Coordination
Phase 1: Preparation (1-2 days before)
- Reduce DNS TTL to 300 seconds
- Initial file sync (while site remains live)
- Database backup and test restoration
- Email account inventory and credential verification
- Test imapsync with a few test accounts
DNS Management Resources:
Phase 2: Pre-Migration (Day of migration)
|
1 2 3 4 5 6 7 8 9 10 |
# 1. Final file sync (expect this to be fast) rsync -avhz --progress /var/www/html/ user@destination:/var/www/html/ # 2. Start email migration (can run parallel) nohup ./migrate_all_emails.sh > email_migration.log 2>&1 & # 3. Monitor email migration progress tail -f email_migration.log |
Phase 3: Maintenance Window
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# 1. Put site in maintenance mode cp maintenance.html /var/www/html/index.html # 2. Stop services systemctl stop apache2 systemctl stop mysql # 3. Final file sync rsync -avhz --progress --delete /var/www/html/ user@destination:/var/www/html/ # 4. Final database dump and transfer mysqldump --single-transaction --skip-lock-tables --all-databases | \ ssh user@destination "mysql" # 5. Update DNS records # Point A records to new server IP |
Phase 4: Post-Migration Validation
- Test website functionality
- Verify database connectivity
- Test email sending/receiving
- Check SSL certificates
- Monitor error logs
Troubleshooting Common Issues
rsync Issues
Permission Problems
|
1 2 3 4 5 6 7 8 |
# Fix ownership after transfer chown -R www-data:www-data /var/www/html/ chmod -R 755 /var/www/html/ # Preserve original permissions during sync rsync -avhz --perms --owner --group /source/ user@dest:/target/ |
Connection Issues
|
1 2 3 4 5 6 7 |
# Test SSH connectivity ssh -vvv user@destination-server.com # Use rsync with verbose SSH rsync -avhz -e "ssh -vvv" /source/ user@dest:/target/ |
Troubleshooting Resources:
Large File Timeouts
|
1 2 3 4 |
# Use compression and bandwidth limiting rsync -avhz --bwlimit=1000 --timeout=300 /source/ user@dest:/target/ |
MySQL Issues
Large File Import Errors
|
1 2 3 4 5 6 7 8 |
# Increase MySQL timeout settings mysql -u root -p -e " SET GLOBAL net_read_timeout=600; SET GLOBAL net_write_timeout=600; SET GLOBAL wait_timeout=600; SET GLOBAL max_allowed_packet=1073741824;" |
Memory Issues During Restore
|
1 2 3 4 |
# Use --single-transaction with --quick for large tables mysqldump --single-transaction --quick --lock-tables=false database_name |
imapsync Issues
SSL Certificate Problems
|
1 2 3 4 5 6 7 |
# Skip SSL verification (use with caution) imapsync --host1 source.com --ssl1 --sslargs1 "SSL_verify_mode=0" \ --host2 dest.com --ssl2 --sslargs2 "SSL_verify_mode=0" \ --user1 user@example.com --password1 'pass1' \ --user2 user@example.com --password2 'pass2' |
Authentication Failures
|
1 2 3 4 5 6 |
# Test IMAP connectivity manually telnet mail.example.com 143 # or for SSL openssl s_client -connect mail.example.com:993 |
Large Mailbox Handling
|
1 2 3 4 5 6 7 |
# Use size limits and folder exclusions imapsync --maxsize 50000000 \ --exclude 'Sent|Drafts' \ --maxmessagespersecond 2 \ (other options...) |
Performance Monitoring
System Resource Monitoring
|
1 2 3 4 5 6 7 8 9 10 |
# Monitor during migration watch -n 1 'iostat -x 1 1; free -h; df -h' # Network usage iftop -i eth0 # Process monitoring htop |
Monitoring Tools:
Migration Progress Tracking
|
1 2 3 4 5 6 7 8 9 10 |
# File transfer progress rsync -avhz --progress --stats /source/ user@dest:/target/ # Database restore progress with pv pv database_dump.sql | mysql -u username -p database_name # Email migration monitoring tail -f email_migration.log | grep "Transfer summary" |
Security Best Practices
SSH Hardening
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Use specific SSH keys for migration ssh-keygen -t ed25519 -f ~/.ssh/migration_key # Add to SSH config cat >> ~/.ssh/config << EOF Host migration-dest HostName destination-server.com User migration-user IdentityFile ~/.ssh/migration_key StrictHostKeyChecking yes EOF |
Database Security
|
1 2 3 4 5 6 7 |
# Create migration-specific MySQL user mysql -u root -p -e " CREATE USER 'migration'@'%' IDENTIFIED BY 'secure_password'; GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON *.* TO 'migration'@'%'; FLUSH PRIVILEGES;" |
Email Security
- Use application-specific passwords where available
- Store passwords in secure files with restricted permissions
- Delete password files after migration
- Enable two-factor authentication post-migration
Post-Migration Checklist
Immediate Verification
- ☐ Website loads correctly on new server
- ☐ Database connections working
- ☐ Email sending/receiving functional
- ☐ SSL certificates valid
- ☐ Log files show no critical errors
24-Hour Monitoring
- ☐ Website performance metrics
- ☐ Database query performance
- ☐ Email delivery rates
- ☐ Error log analysis
- ☐ User feedback collection
1-Week Follow-up
- ☐ Search engine indexing status
- ☐ Email deliverability scores
- ☐ Performance optimization opportunities
- ☐ Backup verification
- ☐ Documentation updates
Recovery Procedures
Rollback Plan
|
1 2 3 4 5 6 7 8 9 10 |
# Quick DNS rollback # Change A records back to original server # Database rollback mysql -u username -p < pre_migration_backup.sql # File rollback (if needed) rsync -avhz --delete /backup/original/ /var/www/html/ |
Disaster Recovery
- Maintain backups on original server for 30 days
- Document all configuration changes
- Keep migration logs for troubleshooting
- Test rollback procedures before migration
Advanced Techniques
Using Screen/Tmux for Long Migrations
|
1 2 3 4 5 6 7 8 |
# Start screen session screen -S migration # Run migration commands # Detach with Ctrl+A, D # Reattach with: screen -r migration |
Session Management:
Parallel Processing
|
1 2 3 4 5 6 7 |
# Multiple rsync processes for different directories rsync -avhz /var/www/site1/ user@dest:/var/www/site1/ & rsync -avhz /var/www/site2/ user@dest:/var/www/site2/ & rsync -avhz /var/www/site3/ user@dest:/var/www/site3/ & wait |
Database Replication Setup
|
1 2 3 4 |
# For zero-downtime migrations, consider setting up temporary replication # This is beyond basic migration but useful for critical systems |
Advanced Migration Techniques:
Conclusion
This migration strategy using rsync, MySQL dump, and imapsync provides a comprehensive approach to website migration. The key to success is thorough planning, incremental transfers, and careful coordination of the maintenance window.
Remember to:
- Test everything in a staging environment first
- Have rollback procedures ready
- Communicate with stakeholders about timelines
- Monitor the migration closely
- Validate thoroughly after completion
For additional support, refer to the official documentation:
