A Scalable Magento 2 Architecture
I write this post with mixed feelings as a company in my portfolio has recently decided to no longer on-board new Magento development projects due to various reasons beyond the scope of this post. However, we are still here to help with companies that are looking for a hand on designing infrastructure, maintaining the systems side of Magento environments, and migrating working Magento instances to the cloud.
For those of you who are not familiar, Magento is one of the most popular E-Commerce platforms used by e-tailers of all sizes. Perhaps it gained its popularity due to the fact that it’s an opensource project with a freely distributed community edition helping lots of companies build out powerful e-commerce websites quickly at extremely low costs. It has also in the past been one of the only few choices in its class that could efficiently handle large amounts of SKUs until the recent years where some of its competitors have caught up a bit. According to Wikipedia, there are over 100,000 stores globally that run on Magento and the software itself has been downloaded more than 2.5 million times. In 2018, Magento was acquired by Adobe for 1.68 billion US dollars, effectively becoming one of the members of Adobe’s brigade of web presence building tools.
As you can tell from the above, even though we always advice customers to build out their e-commerce sites the right way from the start with custom builds rather than using pre-built template based platforms for many very good reasons, it’s extremely easy for budget conscious businesses to be tempted to go with a template based platform to reduce up-front costs instead. Whether this is a good decision or not is yet another topic beyond the scope of this post. However the point is that because of this phenomenon, we have over the years done quite a bit of work with Magento and some of the other platforms like WooCommerce, Odoo, and even Shopify.
Around two years ago, a friend’s company running 3 e-commerce websites on a single Magento 1.9 instance came to me and asked for some help with their site. Apparently, at the time, the site would keep on crashing especially during high season and that was causing them business opportunities. I took a look at it and it was a typical growth situation. Basically, everything was running on a single VPS that just wasn’t cutting it anymore. So without going into too much detail, we essentially decoupled their Magento instance to an AWS EC2 auto-scaling group mounting EFS shares (we talked about containerization but the in-house team wasn’t quite ready for such a big leap) and Aurora to handle the database along with slapping Cloudflare in front of the whole thing to do some CDN and caching. They then had a somewhat scalable infrastructure from there and a single point to upload media and configure magento for all EC2 web instances. Off we went on a launch and the problem was solved.
Fast forward to a few a months ago, Magento 1 using Paypal was falling out of PCI compliance and Paypal urged everyone to upgrade to Magento 2. So we regrouped with the client and discussed a plan of action. This time around, they were ready for containerization and AWS Fargate was at the same time finally mature enough for us to give it a run for its money. The hard part now is to migrate the Magento 1 data to Magento 2. This is the most painful part which I will definitely not cover here but let’s just say that we were able to successfully migrate all of the pertinent data over to a new Magento 2 instance running in Docker after a lot of pain and many sleepless nights. If anyone is thinking about doing this, I would strongly suggest you don’t go cheap and hire a strong firm that’s specifically focused on Magento with lots of experience doing migrations of this type. The difference is night and day.
The New Archtecture:
Now that we have a pretty solid Magento 2 instance running in Docker on our local dev environment with the migrated data from Magento 1, we are ready to throw this onto AWS. We first imported the database to Aurora and created the Redis instance. Then, we started up an EC2 instance called DevOps and ran the image with it to reconfigure it for using Aurora and made some of the directories on EFS to be test mounted by the container. After that, we committed the image and pushed it to a newly created ECR registry.
We are then ready to give it a run on Fargate, we created the cluster and task definitions so that the image we have prepared is launched to have 2 minimum containers with an auto-scaler that maxes out at 4 containers sitting behind an Application Load Balancer which are configs derived from our past traffic based estimates. So why Fargate you might ask? Here is my logic behind it:
- No servers or orchestration infrastructure to manage
- Handles EFS mounting and launching/monitoring of docker containers for you via task definitions
- Load Balancer is automatically adjusted on launch and auto-scaling
- Overall cost reduction to only paying for what’s used compared to EC2
Of course, we reused the bastion host that has OpenVPN (external) and ssh (internal LAN only) enabled to allow our customers to configure Magento and upload images to EFS that persists across all containers. It will also serve as the gateway for the client to access DevOps for making docker image changes and accessing the new analytics system we will be installing after.
As mentioned, the client wanted to run some analytics on the data in Magento so it was only logical to create a read replica of the production database so that they can run an EC2 instance and install whatever they wanted to grab data and analyze with. So that’s exactly what we did.
And that’s it! We logged into Cloudflare, swapped out the CNAME of the domains to the new load balancer and went live. This sums up all the components of a modern scalable Magento 2 architecture as diagrammed above. There are more improvements that can be done or even components that’s already there which I didn’t really talk much about like varnish and maybe even CI/CD but what’s clearly within the diagram alone is already a solid enough foundation for general scaling and growth and definitely ready for you to add more or improve upon without any unwanted architectural issues.
If there are any questions or suggestions, please feel free to drop a comment below or get directly in touch!