Moving our CDN (10+ GB images) to a new server

Today, we moved our CDN from AWS to DigitalOcean. There were a few reasons for this.

So, we moved our CDN, which had 10GB images saved on the disk, to DigitalOcean Frankfurt data center in about an hour. The process is really simple than you might think.

Make a plan

First, I wrote down all the steps I have to do in a Notion page.

I'll explain each step.

Setting up the New Server

DigitalOcean droplets are easy to create and manage. Here, I choose to save the images in a DigitalOcean Volume. A volume is like a hard drive that you can attach to the droplet. The reason to choose Volumes is that we can increase the size of storage without upgrading the main server. Usually, CDNs don't require a lot of CPU power. It's all about processing and serving each image one time. We use Cloudflare for caching. So, the Cloudflare cache will serve the cached image automatically without requiring our servers to process it again. So, we chose a Droplet with 1GB RAM ($5/month) and attached a 100GB Volume ($10/month) to it.

Then, we installed our CDN application, which is written in PHP. It handles image uploading and delivery.

Server to server SSH

This is the most interesting part for me.

If you don't love SSH, you haven't used it much.

I use Git Bash for SSH connections. You can use anything like Putty or Browser extensions. Log into both of your servers in two terminals.

Go to the new server's SSH session. Check the .ssh folder if you have any ssh keys generated (id_rsa and id_rsa.pub or a similar file). If you don't have any, use this command to generate one. You don't really need to add a passphrase.

ssh-keygen -t rsa

Next, open the file and copy the content from your terminal to your computer's clipboard. In Git Bash, you can select the text, right-click, and copy.

The command to display the public key.

cat id_rsa.pub

Now, you want to paste the copied key to the old server's authorized_keys file.

Switch to the other SSH session (old server). Then, open .ssh/authorized_key file for editing. I prefer vi but you can use vim or nano.

vi ~/.ssh/authorzied_keys

Now, paste the copied key here. Make sure to add a line break if you have any other keys. Then, save the changes. Now you can SSH from new server to the old server, which makes it possible to use rsync.

Copying Files

There are two ways to copy files via SSH: scp (Secure Copy) and rsync. I first started copying a small portion of data to the new server with both methods to test the speed. rsync was a lot faster than scp . Here's a good explanation on scp vs rsync. rsync definitely has more optimizations.

Let's copy the content now. Run the following rsync command in the new server's SSH session.

rsync -azP [email protected]:/path/in/old/host/ /path/in/new/host

Note the ending / in [email protected]:/path/in/old/host/. It instructs rsync to copy the contents of that folder to the destination, not the folder itself.

In my case, it only took a few minutes to copy 10GB images. But, it will depend on your network speed.

Changing DNS

If you migrate a live website, you'll need to change DNS records too. Here's what I did.

That's all. I thought it would be a daunting process to migrate large files from a server to another server. But, that's pretty simple if you know the basic SSH.

If you have any questions, feel free to comment below.

Tagged: devops
You can connect with me on Twitter or Linkedin.
Latest on My Blog
PHP Beginner's Tutorial
Beginner's PHP Tutorial
Image for Resizing Droplets: A Personal Experience
Resizing Droplets: A Personal Experience
Image for Disqus, the dark commenting system
Disqus, the dark commenting system
Image for Creating a Real-Time Chat App with PHP and Node.js
Creating a Real-Time Chat App with PHP and Node.js
Image for All About MYSQLI Prepared Statements in PHP
All About MYSQLI Prepared Statements in PHP
Image for Image Upload with AJAX, PHP, and MYSQL - The Beginner's Guide
Image Upload with AJAX, PHP, and MYSQL - The Beginner's Guide
Related Articles
72