Idealstack2022-01-19T17:33:27+13:00https://idealstack.io/blog/feed.atomhttps://idealstack.io/blog/new-features-ubuntu-1804-supportNew Features: Ubuntu 18.04 support2019-06-06T12:00:00+12:002019-06-06T12:00:00+12:00IdealstackWe've now released (optional) support for hosting on ubuntu 18.04. Your existing sites won't change but new sites you create default to 18.04 and you can choose to update your existing sites by editing the Hosting Plan. Ubuntu 16.04 continues to receive security updates until January 2021...We've now released (optional) support for hosting on ubuntu 18.04.  Until recently our hosting images were based on Ubuntu 16.04, the previous long term support release of Ubuntu, but now we've completed the necessary integration work and testing to support 18.04 as well. In practical terms there aren't a lot of differences between the versions..The versions of core apps have all be updated (PHP versions are updated independently of the operating system in Idealstack).  Probably the biggest difference is the use of a new version of Apache .  There may be minor performance differences between the versions although we don't see any in our testing. Your existing sites won't change but new sites you create default to 18.04 and you can choose to update your existing sites by editing the Hosting Plan.  Ubuntu 16.04 continues to receive security updates until January 2021 so you don't need to rush to do this, we will release more updates about how we will automatically port your sites to 18.04 closer to the end of life for 16.04 \]\]>https://idealstack.io/blog/new-features-site-metrics-streamlined-ses-managementNew Features: Site metrics, streamlined SES management2019-05-26T12:00:00+12:002019-05-26T12:00:00+12:00IdealstackNew features we've released in May : better diagnostics and metrics for sites, streamlining the management of SES for email...In may we released a few new features to make it even easier to manage sites in idealstack # Site Metrics and Diagnostics When editing a site in idealstack, check out the new 'Status' tab This page gives you a number of useful features.  Firstly the diagnostic section, which lets you quickly check your site is healthy This checks that there are healthy hosts for your site (ie that the servers it is running on are return OK on the healthcheck) and that there are no 500 errors being returned by the site (eg, due to code problems) This page also shows a number of graphs of site statistics.  For each of these, there's a link to AWS cloudwatch where you can do more filtering, view other date ranges and so on for this data A brief summary of the hits appears as a sparkline on the dashboard and the diagnostic status is also shown there: # Improvements to SES management When setting up email in your hosting plan, you can now set an SES region and see the verification and sandbox status directly from idealstack. On the site Connect tab you can see the SMTP details for your site If you fold open the _Troubleshooting_ dropdown now you can also send a test email \]\]>https://idealstack.io/blog/new-features-web-based-file-managerNew Features - Web-based file manager2019-05-25T12:00:00+12:002019-05-25T12:00:00+12:00IdealstackIdealstack has a new feature, a web based filemanager that lets you edit, upload and download files without needing an SSH connection (great for making changes on the go)...We've just released a new feature that we think people will find quite useful - a web based file manager. Now you don't need to connect to SFTP to make file edits or manage files - this is really handy if you need to make quick changes on the go and don't have your SSH key or an SFTP client. ### Features - create/edit files and directories - copy/move files and directories - download files and directories - upload files directly, via URL or with drag & drop - extract archives (tar, tgz, tar.gz, tar.bz2, zip) - change permissions - image preview ### How to access it Access the filemanager under the 'Connect' tab for your site.  How it works is pretty self-explanatory \]\]>https://idealstack.io/blog/faster-dependency-free-php-sessions-dynamodbFaster, dependency free PHP sessions in dynamodb2019-05-10T12:00:00+12:002019-05-10T12:00:00+12:00IdealstackWe've released the session handler that we use to enable transparently, automatically storing sessions in DynamoDB...Dependency-free DynamoDB sessions One of the great features of the Idealstack platform is the way we automatically setup PHP to use DynamoDB to store sessions, rather than PHP's default file-based session storage.  It's a big part of how we can [automatically cluster-enable generic PHP apps like wordpress](/content/how-it-works/index.html). We started off using the official AWS SDK's session handler for this, but found some issues with that.  So we've created our own session handler and released it on github under an Apache 2.0 license (the same open source license as the AWS PHP SDK). # Why DynamoDB? If you are going to run PHP in a clustered environment, PHP's default file-based session handler won't work - sessions aren't shared between the different servers that might be running the website, so users will lose their session every time the load balancer sends them to a different server. There's a few well-established cluster-friendly ways to do sessions in PHP.   Popular solutions are to use an SQL database, or to use Memcached or Redis.  But all of these require an additional server and none of them scale as well as DynamoDB does.  With DynamoDB you can pay for the capacity you use but then scale up to any capacity you require.  We feel that if you are hosting PHP on AWS, dynamodb sessions are the best solution. # Why dependency-free? The [PHP SDK provides a session handler](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/service_dynamodb-session-handler.html) that uses DynamoDB for PHP sessions.  For a while we used this to handle sessions for our clients and it works well. There's obviously benefits to using the officially supported AWS codebase if you can. The problem though is that the AWS SDK is a beast.  It has a number of dependencies (things like Guzzle) and a very elegent, abstracted code structure - but therefore also a complex one.  Using this for sessions means a large number of PHP files need to be included in every page load.  This is fine if you are using the PHP SDK in other places in your code - you need to load these files sooner or later anyway.  But if you are using a common CMS system like wordpress you may never use these libraries elsewhere, so you end out using up memory, opcache caching space and disk IO loading these files for nothing. The other problem you face - which admittedly may be a little specific to the quirks of how we are running it - is that because we are injecting this session handler code into other random code, you can get conflicts between versions for the libraries it uses.  If the code you are running depends on a different version of Guzzle or the PHP SDK (Guzzle in particular has a lot of incompatible versions), then weird bugs may happen.  The SDK also requires an autoloader which can clash with your code's own autoloading.  So in general it's better if we can use a session handler that doesn't rely on external libraries or autoloading. # Dependency-free session handler for dynamodb For these reasons we built our own session handler from scratch with no dependencies, and we are releasing it under an open source Apache2 license.  [Grab the code from github](https://github.com/Idealstack/dynamodb-sessions-dependency-free). - Essentially a drop-in replacement for the official session handler in the AWS SDK - Dependency-free - does not depend on any other composer packages. Only requires the core curl and json extensions be enabled in PHP - Does not require an autoloader (although will work fine with one, eg composer) - Supports most common AWS authentication methods (eg instance profiles, ECS task roles, .aws config files, environment variables) - Compatible with all major PHP versions (even PHP 5.6, for all you luddites out there) - As a nice bonus, it's also about 30% faster # How to use it Configuration is identical to the session handler in the official SDK, so if you are already using that it's a drop-in replacement. 1. ### Create a table in dynamodb to store your sessions - Set the primary key to 'id' - Once the table is created enable TTL.  Note the AWS docs don't tell you to do this (we've opened a github [pull request](https://github.com/awsdocs/aws-php-developers-guide/pull/38) suggesting doc updates about this) but it's a good idea even with the official SDK session handler.  Otherwise you'd need to garbage collect sessions which consumes read/write capacity and slows things down (and costs) - - Set the TTL attribute to 'expires' - Consider using the 'on demand' capacity mode.  By default dynamodb sets up provisioned capacity with autoscaling.  This may (or may not) be cheaper but it means your site will 'go slow' or possibly timeout as it nears the capacity limits until autoscaling kicks in.  'On demand' scales instantly and automatically.  It's not eligible for the free tier though if you care about that. 2. ### Setup the session handler in your code - You can install the session handler using composer, but composer and it's autoloader is not required as they are with the SDK - Create AWS credentials with access to the table.  We recomend you create an IAM policy and associate it as an instance role or ECS task role, the [AWS SDK docs tell you the minimum required permissions](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/service_dynamodb-session-handler.html#required-iam-permissions).  You can also save the credentials in ~/.aws/credentials or use something like [PHP dotenv](https://github.com/vlucas/phpdotenv) to store them. - Create the session handler object and call 'register' on it: \`\`\`php use Idealstack\\DynamoDbSessionHandlerDependencyFree; // or if you don't want to use composer auto-loader, try: // require( **DIR** .'/vendor/idealstack/dynamodb-session-handler-dependency-free/src/DynamoDbSessionHandler.php'); (new Idealstack\\DynamoDbSessionHandlerDependencyFree\\DynamoDbSessionHandler( \[\ 'table\_name' => 'your-session-table-name',\ // Credentials. In production we recomend you use an instance role so you do not need to hardcode these.\ // At least make sure you don't hardcode them and commit them to github!\ 'credentials' => \[\ 'key' => 'AAAAAAAAAAAAAAAAAAAAAA',\ 'secret' => 'BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'\ \],\ // These are all defaults.\ // // Base64 encode data when reading and writing. Avoids problems with binary data, Note this is\ // // not the behaviour of the AWS SDK, so set to false if you require compatibility with existing\ // // sessions created with the SDK\ // 'base64' => true,\ // 'hash\_key' => 'id',\ //\ // // The lifetime of an inactive session before it should be garbage collected. If it isn't provided,\ // // the actual lifetime value that will be used is ini\_get('session.gc\_maxlifetime').\ // 'session\_lifetime' => 1440, // 24 minutes\ // 'consistent\_reads' => true, //You almost certainly want this to be true\ // 'session\_locking' => false, //True is not supported\ \] ))->register(); ```` 3. ### Test that it's working * This code will create a session.   Refresh it a few times to confirm that the timestamp it outputs doesn't change.  You can also look in the dynamodb table to confirm it is creating new records with session data ```php if (session_module_name() != 'user') { throw new \Exception("not using session handler"); } session_start(); if (! array_key_exists("test", $_SESSION)) $_SESSION["test"] = microtime(); echo "OK ".$_SESSION["test"] ```` 1. ### Make sure your app actually uses native sessions - One thing to be aware of is that a lot of PHP apps and frameworks override PHP's native sessions with their own session handlers.  Most of these use files to store sessions anyway, which is what we are trying to avoid.  To make these apps use your sessions you'll need to find a way to make them use native sessions.  Typically there is a module available for this in common frameworks like Laravel or Symfony.  We've documented steps for many popular  frameworks, apps and CMS's in our [app setup instructions](/content/help/tips/how-install-popular-php-apps-aws-using-idealstack/index.html) \]\]>https://idealstack.io/blog/using-awss-new-aws-backups-tool-backup-phpmysql-websites-idealstackUsing AWS's new AWS Backups tool to backup PHP/Mysql websites on Idealstack2019-01-23T13:00:00+13:002019-01-23T13:00:00+13:00IdealstackUsing AWS's new AWS Backups tool to backup PHP/Mysql websites on Idealstack...AWS have [released a new tool, AWS Backup](https://www.google.com/url?q=https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-aws-backup/&sa=D&ust=1548208628470000), which is quite useful for backing up PHP-based websites hosted on Idealstack or any other solution that uses EFS for file storage (eg most of AWS’s reference architectures for platforms such as Wordpress use EFS as well). Until recently backup was a feature missing from EFS and you were forced to implement your own solutions for backing up EFS data.   Now that there is an official solution from AWS  we recommend this over our [previous solution of using Duplicity to backup to S3](/content/blog/ecs-efs-s3-backups-using-duplicity/index.html) There’s a lot of benefits of using AWS Backup over a roll-your-own solution like duplicity: - It takes a snapshot of the data, so things will be consistent - It doesn’t slow down or interrupt the operation of your filesystem and database while the backup is happening - It’s a solution managed by AWS, so you can be more confident it won’t break (which is a big problem with backups!) - For services like RDS which already had good solutions for backup, AWS Backup lets you have all your backups in one place which makes them easier to manage and easier to verify that they’re configured how you want.  AWS Backup still uses the same methods the service itself uses (eg RDS Snapshots) so you can still access your AWS Backups directly within RDS itself as well as through AWS Backup In this article we walk you through setting up backups in AWS Backup and the process of restoring backups for stacks managed Idealstack. The same steps should also work if you don’t use Idealstack, with a few small tweaks. ## Setup AWS Backup First go to _AWS Backup_ in the AWS menu.  Make sure you do this in the same AWS region that your stack is in. And create a new backup plan On the next screen you are asked to choose the times and retention policies for your backups.  It’s easiest to start from AWS’s examples For backing up EFS volumes like Idealstack uses, cold storage is a fifth of the price of warm storage (see [the pricing](https://www.google.com/url?q=https://aws.amazon.com/backup/pricing/&sa=D&ust=1548208628472000)).  So unless you restore your backups a lot I prefer to edit the plan and change to use cold storage after 2 days (I find that 99% of the time when you restore a backup you want the most recent backup) Now you need to assign resources to be backed up. If you have your own stack you need to think about what resources you backup, but if you use Idealstack it is fairly simple - all your data is stored on EFS (for files) and RDS (for databases).  So click Assign Resources.  Then give your resource assignments a name Now under the assign resources section, select the EFS and RDS resources If you have multiple stacks, it may be hard to find the resource ids - you can get these from the Stack Layout under the Status tab for your stack in Idealstack.  Clicking the RDS and EFS icons in this layout will take you to these resources in AWS Obviously, if you use other resources like DynamoDB managed outside of Idealstack  you may also want to add these here. Then click ‘Assign Resources’ to save ## Restoring Backups To restore a backup, go to ‘Protected Resources’ and find the resource you want to restore.  Depending on what was deleted this could be either EFS (if you need to restore a file) or RDS (if you need to restore data out of the database) ### To restore an RDS Database Click on the RDS resource under Protected Resources and choose the backup you want to restore Then click the \_Restore \_button This process will create a new database instance based on the original.  Most of the defaults on the next screen are sensible, except that because this database is only temporary and you are going to destroy it when you are finished with it, most of the time it makes sense to tweak the instance class, storage type and multi-az settings to cheaper options: Give it a name Under the subnet group and VPC settings, make sure you select the VPC for your stack - copy what is set for the original instance Then click Restore Backup Now you will see the restore job happening Wait for this restore job to complete ### Copy databases or data back to your production site Go to the RDS console and find the restored database instance. First you must modify it to use the correct security group.  Match the one in use by your live database And on the next screen make sure you select _Apply immediately_ Then find the endpoint for your restored database Now [SSH to one of your sites in idealstack](https://www.google.com/url?q=/help/configuration/connect-your-site-ssh&sa=D&ust=1548208628477000). You should be able to connect to your database using the username and password of your live database (you can find this in the Stack in idealstack) Depending what you are trying to achieve you may now want to move this data to your live server.  You can do this with mysqldump.  For instance like this (replace the users/databases/hostnames with your ones) ``` mysqladmin -h yourdatabase -u youruser -p create restored_database mysqldump -h restored-backup.ci0wdhe0flzq.us-west-2.rds.amazonaws.com -u youruser-p yourdatabase | mysql -h yourdatabase -u youruser -p restored_database ``` Once you are finished with the temporary restored instance, delete it in the RDS console so you don’t need to keep paying for it ## Restoring Files from the EFS Under the ‘Protected Resources’ screen click on your EFS Choose the recovery point you want to restore and click Restore On the next screen ensure ‘Restore to directory in source file system’ is selected Now wait for the restore job to complete _Note: at the time of writing this, the restore jobs are inexplicably slow - taking over an hour to complete even with only 25GB of data in my tests.  Perhaps this is due to the new nature of the service and may improve._ ### Copying the files back to your production site Now you need to restore the files from the subdirectory on the EFS that they were restored to.  To do this you need to login to one of your EC2 instances and copy the files Under EC2 find your instance and copy it’s Public DNS If you have a lot of instances, In Idealstack you can find this in the Stack Layout under _Stack Status_\- clicking the instance will take you directly to it in the AWS console. Once you’ve done this SSH to the instance as the user ‘ec2-user’. Note that if you have ssh’d to this machine before it’s host key may have changed causing ssh to give you an error.  Editing ~/.ssh/known\_hosts or running ssh -R should fix that The EFS is mounted under _/mnt/efs_ \_\_ You will find the restored data under a new directory with a name like ‘ _aws-backup-restore-$restore\_date_’ eg like so: You can poke around in here to see the data included in the restore. If you want to restore the files of an individual site, use rsync to copy them back.  First install rsync (it isn’t included by default) ``` sudo yum install rsync ``` Idealstack uses a data structure on the EFS that puts each site’s files into a directory under ‘ _idealstack-sites/raw/_’ To restore the files of an individual site into a subdirectory within that site use something like this: ``` rsync /mnt/efs/aws-backup-restore_2019-01-23T00-34-14-054Z/idealstack-sites/raw/8600219472084598811/ /mnt/efs/idea lstack-sites/raw/8600219472084598811/restored-data ``` The long number _8600219472084598811_ is the site id.  You can find it by editing the site in idealstack and copying it out of the url (eg [https://app.idealstack.io/sites/8600219472084598811/edit](https://www.google.com/url?q=https://app.idealstack.io/sites/8600219472084598811/edit&sa=D&ust=1548208628482000)).  Of course if you just want to overwrite everything with the data from the backup (are you sure? This is often not a good idea) then use ``` rsync /mnt/efs/aws-backup-restore_2019-01-23T00-34-14-054Z/ /mnt/efs ``` \]\]>https://idealstack.io/blog/how-host-php-websites-aws-cloud-comparing-major-optionsHow to host PHP websites on the AWS cloud - comparing the major options2018-06-18T12:00:00+12:002018-06-18T12:00:00+12:00IdealstackComparing the options for hosting PHP based sites and apps on AWS: build something yourself, Elastic Beanstalk, Lightsail or Idealstack?...All the ways to host PHP websites and Apps on the Amazon AWS Cloud - compared PHP is one of the most popular languages for building websites and web applications.  It’s fast and easy, there’s a thriving community of developers, and it has an excellent base of applications and frameworks like Wordpress and Laravel to work with.  All this usually means PHP is the quickest and most affordable way to build a website or web app. Unfortunately the world of PHP hosting remains remarkably backward.  Many PHP sites and apps are still hosted on a single server with no fault tolerance or scalability.  The cloud revolution that has completely changed the face of most other parts of IT has largely passed PHP by. Why is that? Well it’s been pretty hard to get a PHP app working well on a scalable cloud cluster.  This has often driven people to use other languages such as Python, Ruby on rails or Nodejs to build the kind of apps that need scalability and reliability. In fact many people think PHP can’t be used to build these kinds of apps! (which is nonsense).  But it needn’t be this way, and there’s a number of ways to get a PHP site up and running on AWS # Get a single EC2 instance and run Linux on it One option is to setup an EC2 instance, which essentially just provides a virtual machine.  You can of course treat this like any other server, for instance by installing a LAMP stack on it and running PHP sites, or even installing a hosting console like Plesk or CPanel to make setup and maintenance a bit easier. The downside to this is that you're not really any better off than running your own server.  You don’t have to worry about hardware anymore, but you’ve still got a little server for which you have to manage OS updates, security and more.  It’s really not worth doing this. # Elastic Beanstalk Probably a better option for many is to use Elastic Beanstalk.  EB simplifies the task of creating a ‘proper’ stack on AWS. In Elastic Beanstalk you upload your app and choose a platform (eg PHP) and EB will do the rest. The big benefit of this is that what Elastic Beanstalk builds is a ‘well architected’ cloud platform.  It’s not just some simulating some kind of retro solution from the past like a single EC2 instance. You can enable ‘Managed Updates’ which provide operating system updates without you needing to manage these, which is a big win for security and easy of maintenance. Probably the biggest con is that EB doesn’t do shared hosting.  Each “App” needs its own complete cluster of multiple instances and other services in order to provide a scalable and reliable platform.  This means it’s not cost effective for hosting websites (apart from extremely large ones) as it will cost hundreds of dollars per month. The other is that it only works well with modern apps custom built for cloud environments : eg not wordpress, drupal etc.  Unless your app is designed to work with AWS, eg using S3 for file storage, database shared sessions etc, it’s not going to work without a lot of extra tweaking.  One can’t simply upload wordpress to EB and expect it to work. Your developers will need to change their workflow to use EB.  You need to upload your app as a ‘bundle’, and you can’t access the live environment to make direct code changes. What all of this means is that EB isn’t a good option for hosting and developing websites. It might be a good option for apps (eg if you’re a startup building the next Uber or Netflix). # Build your own scalable hosting cluster This option is pursued by many startups, and it makes sense if you have a lot of sophisticated devops developers, a lot of funding and you know you’re in it for the long term. If you go down this route, I recommend you take a look at [Cloudformation](https://aws.amazon.com/cloudformation/) or possibly [Terraform](https://www.terraform.io/)  as a way to create ‘infrastructure as code’.  Clicking around and manually creating things in the AWS web console is a great to learn how things work ‘under the hood’, but it’s not a practical way to run an infrastructure and in many cases the time you spent learning in the GUI would be better spent learning Terraform or CloudFormation. The biggest downside to this is that it takes a very long time.  Learning all the necessary tools and putting them together will probably take between six months and a year, plus then a period of several years of enhancing your stack and working around problems as you discover them.  After many years you will hopefully have the sort of stack that you hear about Google, Facebook or Netflix having, but the years of instability, learning and pain along the way has an incalculable cost. # Lightsail Lightsail is a relatively new AWS service that provides a VPS (Virtual Private Server), similar to those provided by the likes of Digital Ocean, Vultr or many other providers. Pricing is pretty cheap, starting at $5 USD per month (although realistically expect to pay $10 USD for something production-ready), and setup is pretty simple too.  Lightsail comes closest in the AWS cloud to a service built for PHP developers One problem lightsail solves is providing secure affordable hosting.  By giving each of your sites a seperate lightsail instance they are isolated from each other. Lightsail isn’t ‘real AWS’ - it doesn’t really provide any scalability or fault tolerance.  Lightsail provides a simple load balancer - so if you create multiple Lightsail VPS’s, install your app on all of them, and work out how to make it operate on a cluster you can distribute traffic over lightsail instances. This of course increases the cost of hosting your app considerably, but you still have to do much the same work as building your own cluster to really make this work. We wouldn’t recomend this approach if you truly wanted to make a fault-tolerant scalable architecture for an app . Similar to installing an EC2 instance, a VPS is essentially a little server that needs you to manage it’s security, particularly by installing updates to it’s underlying OS on a regular basis.  That of course is relatively simple provided you know linux, but these machines are not zero-maintenance. # Other third party solutions that ‘use AWS’ There’s a few providers out there that claim to help you setup your app on AWS, but they don’t work on your AWS account - they may allow you to select an AWS instance and region but they’re essentially reselling instances out of their own AWS account.  We won’t name names here, beyond saying that we think this solution is pretty crappy.  You aren't really setting up sites on AWS unless you have access to your own AWS account Often these providers can match AWS’s on-demand pricing - by buying reserved instances on AWS you can reduce costs by 70% and then reselling them to you provides a good markup for them.  Of course you should have the opportunity to signup reserved instances yourself and pocket those savings. These solutions are not ‘really AWS’ unless you can plug in all the other AWS services - eg CloudFront for CDN, Elasticache for caching, services like SQS and SNS for batching and queueing, S3 for affordable enterprise storage.  They often don’t even provide any HA or autoscale. # Idealstack Of course we’re biased, but we think Idealstack is the best way to run PHP on AWS.  Of course we do, that’s why we built it! There’s a number of key reasons for this - some of the other ways of running PHP on AWS can achieve one or two of them, Idealstack is the only system that can do them all: ### You own your hosting Idealstack sets up a stack on your own AWS account, and you own that stack.  Even if you later choose to cancel Idealstack your stack will keep running (although of course you lose access to benefits we provide like automatic updates, and our management interface) ### Developers should focus on building sites, not Devops The first thing that Idealstack does is provide a ‘normal’ PHP environment.  What we mean by that is Idealstack provides the things PHP developers expect: - Upload files over SFTP - SSH to your site and run commands - GUI for database management and access You developers can work with Idealstack just like they do with any other hosting environment, they don’t need to be retrained.  This is great especially when your hiring or dealing with external teams. Setting up the hosting cluster is also very easy and usually takes about 30 minutes, 20 of which you can spend making a cup of tea while the Idealstack system automates everything for you. Idealstack takes away most of the hassles of server management.  In Idealstack servers (ie EC2 Instances) are ‘cattle not pets’. If they misbehave they are automatically replaced.  OS updates are applied automatically by destroying the instance and creating a new one. You can scale them up or down, add new ones manually or automatically, without ever really having to think about them. ### Runs any PHP app Idealstack runs any app like Wordpress, Drupal, Joomla, frameworks like Laravel or CodeIgniter, or your own legacy code on a scalable fault-tolerant cluster, but without you having to modify the app.  How do we do this? - Shared storage by default - using AWS’s EFS shared filesystem, code and fiels are shared across the cluster, so apps that locally write files (which is pretty much all apps) work - Shared sessions by default - Idealstack sets up a DynamoDB table and session handler that writes to this for each site.  This solves one of the biggest challenges of scaling PHP across a cluster - by default PHP sessions are written to files on local disk ### Secure shared hosting In the web industry - we’re often dealing with multiple clients and we want to host their websites affordably.  What that often means though is that we make their hosting a bit crappy. One of the great things about Idealstack is we can flip that equation on it’s head - by providing shared hosting, we can make their hosting great.  The reason is that by combining together a lot of clients you can achieve economies of scale that mean you can have a fault-tolerant autoscaling cluster that all these sites run across.  This means you can provide a well-architected high-end hosting environment for a very small cost (as little as $5/site) To achieve this Idealstack runs each site inside a container across a well-architected cluster.  Idealstack uses Docker (with an image we manage) and ECS to do this, but you never really have to worry about it, it happens ‘under the hood’ ### Well-architected, stable secure hosting Unlike most of the other ways of hosting sites on AWS, Idealstack is designed from the ground up exclusively for the AWS platform and it does things ‘the right way’.  What we mean by that is that Idealstack runs a cluster across multiple availability zones in AWS, and provides autoscaling to deal with load spikes (load spikes are the number 1 reason PHP websites experience outages). AWS services are used automatically where possible - for instance ACM is used for registering SSL certificates, SES for sending email, ALB for load balancing, EFS for storage, RDS for database servers, ECS for managing clusters. You can plug in other AWS services like CloudFront for CDN, Elasticache for caching, SQS for queuing, S3 for mass storage and so forth. # Conclusion So obviously we think Idealstack is the ideal stack for hosting PHP on AWS. Of course, you say, they're biased! They would say that!  But that's why we built it and what gets us out of bed every morning to work on it.  We think you should give it a try : there's a free trial that you can combine with the AWS free tier to give you a no-risk way of seeing if you agree with our assesment here.  Give it a go! \]\]>https://idealstack.io/blog/ecs-efs-s3-backups-using-duplicityBackups on ECS/EFS to S3 using Duplicity2018-01-29T13:00:00+13:002018-01-29T13:00:00+13:00IdealstackHow to do backups to S3 when you are using ECS with EFS to persist your data....UPDATE: use [AWS Backup](/content/blog/using-awss-new-aws-backups-tool-backup-phpmysql-websites-idealstack/index.html) instead AWS have released a new solution, AWS Backup, which is probably a better choice for most people looking to backup EFS.  We've written an article on [how to set it up](/content/blog/using-awss-new-aws-backups-tool-backup-phpmysql-websites-idealstack/index.html).  We'll keep this doc here though as it might still be useful to those for whom the AWS solution doesn't work well. # Backups on ECS/EFS using duplicity At Idealstack we use EFS a lot as a persistent data store for docker containers on ECS, inspired by [this](https://aws.amazon.com/blogs/compute/using-amazon-efs-to-persist-data-from-amazon-ecs-containers/). But how do you deal with backups? We don’t really want to have to maintain scripts on an EC2 instance - our EC2 instances are unmodified copies of the Amazon ECS Optimised AMI that are created and destroyed all the time and we treat them as cattle, not pets AWS’s recomended solution is backup from EFS to another EFS using [bunch of lambda functions](https://docs.aws.amazon.com/efs/latest/ug/efs-backup.html) that create EC2 instances to do a backup, but we don’t like that - EFS is the most expensive storage product that AWS offers, and we don’t feel it’s suited to long-term archives like backups (on the other hand, EFS to EFS backup will give faster restores than the solution we offer here - there's no reason not to do both if you need both quick restores and long term retention). Our favoured backup system is duplicity. It can backup to S3 (or almost anything else). We then automatically migrate backups to Glacier after 3 months to provide super-long-term retention of everything. So it makes sense to use a docker container running duplicity to do backups. We feel this is a more elegant solution for ECS users than a lot of lambda functions and EC2 instances, and it’s probably cheaper to, using as it does resources you already have running. Note that this solution will work perfectly for users of the idealstack hosting system (it's what we use it for) but it should also work fine for anyone else running ECS & EFS together  - just update the paths we give here with those you use. ## Setting up backups 1. Create an S3 bucket to store your backups. Use the same region as your ECS cluster lives in. Otherwise you can accept all the defaults. 2. Create an IAM policy allowing access to this bucket 1. Go to IAM and navigate to Roles in the IAM menu 2. Click “Create Policy” at the top of the screen 3. On the IAM policy editor, select JSON and paste in the following: [https://gist.github.com/jonathonsim/581dca42a0779a2416aaf1936e4679fd](https://gist.github.com/jonathonsim/581dca42a0779a2416aaf1936e4679fd) 1. Update the s3 bucket name to match yours 4. Click Review policy 5. Give the policy a name (eg S3BackupAccessForDuplicity) 3. Attach this policy to the instance role for ECS. Note that in theory we should use a task role for this, but at this time that doesn’t appear to work in duplicity 1. In ECS - click on your cluster, go to the ECS Instances tab 2. Click your EC2 Instance 3. Under the _Description_ of the instance click the IAM Role 4. Attach the policy you just created to this role : 4. Back In ECS create a task definition: 1. Give it a sensible name 1. Scroll down to _Volumes_ (skip the other settings for now, we’ll come back to them) and click _Add Volume_ __ 1. Add a volume for the data you are backing up 2. Add a volume for the duplicity cache, somewhere persistent (ie on the EFS) 1. Under _Container Definitions_: click “Add Container” 1. Give the container a name, eg DuplicityBackup 2. For the image, we’re going to use [https://hub.docker.com/r/wernight/duplicity/](https://hub.docker.com/r/wernight/duplicity/), so enter “wernight/duplicity” 3. For memory limit - choose a hard limit of 512MB (you don’t want backups using up all your RAM) 4. This section should look like this now: 5. Under environment, enter the command : update to match your bucket. Also think about what the ‘full-if-older-than’ option should be - it depends how much your data changes: /bin/sh, -c, duplicity --exclude=/data/duplicity-cache --exclude=/data/duplicity-restore --full-if-older-than=3M --s3-use-new-style --asynchronous-upload /data s3+http:// **my-example-bucket**/backups; duplicity cleanup --s3-use-new-style --force s3+http:// **my-example-bucket**/backups 6. Setup an environment varialbe with your passphrase. You can generate one here: https://passwordsgenerator.net/ 1. Create an env variable called PASSPHRASE and set your passphrase 7. The only other settings you need to setup are under _Storage_: 1. Add a mount point for the cache, on /home/duplicity 2. Add a mount point for the data on /data. You can check ‘read only’ on this: 3. Choose the log driver of your preference, in production we use awslogs, but we’ll use Syslog for this example as it’s easy : log messages will then appear in the syslog (/var/log/messages on Amazon Linux) 4. Under security - choose the user to run the backup as. Note that it’s great if your file permissions are such that an unpriviliged user can be used here, but that’s not always practical and you may need to choose ‘root’ 8. Leave all the other settings and click _Update_ __ 2. Now click Create on the task definition 7. Now let’s do a backup to see if it works. Under _Actions_, run the task: 1. It should take a while, depending how much data you have. You’ll see items start to appear in your s3 bucket. If it doesn’t work, check the syslog on the instance (or wherever else you configured your logs to go) ## Scheduling your backup to run every night You’ll want to schedule your backup to run every night. This is easy to do as a scheduled task in ECS 1. In ECS, click on your cluster and choose _Scheduled Tasks_ and click _Create_ __ 2. Give your schedule rule a name 3. Choose “ _Cron Expression_” as the rule type 4. Enter an AWS Schedule expression for whent to run the backup. As an example, this will run the backup at 2am every morning cron(0 2 \* \* ? \*) 5. Under _Schedule Targets_ \- give you target an ID, and select your task definition that you created above 6. Accept all the other defaults and click _Create_ __ 7. Things will crunch for a moment before telling you that your scheduled task has been created ## Restoring your backups To restore backups, you can run duplicity on any host, but it can be convenient to create an ECS Task definition for it - then just run that in the ECS console or CLI to restore. Doing this is very similar to the steps above: 1. Create a task definition as above and give it a name (eg DuplicityRestore) 2. Add the location you want to restore to as a volume. We restore onto a directory on the EFS so that there’s space, but use a subdirectory as we don’t want to be nuking our files with restores: 1. Add a volume for the restore directory and call it data.  Here we use /mnt/efs/duplicity-restore 2. Add a volume for the duplicity cache, identical to the one you used in the backup task 3. Add the container 1. Use the same image _wernight/duplicity_ 2. For the command use this, replacing your bucket name: duplicity, restore, --s3-use-new-style,s3+http:// **my-example-bucket**/backups,/data 3. Under Env Variables, create one called PASSPHRASE and paste in the same passphrase that you used in the backup task 4. Under volumes : select the data volume and mount it to /data, select the cache volume and mount it to _/home/duplicity_ 5. Choose the user (eg ‘root’) 4. Save the task definition 5. Run the task. Hopefully you should see your data restored into the target directory you chose ## What if I want to restore from further back? When running the task, under Advanced Options & Container Overrides, modify the command and add **--time=3D** (or whatever date specification you want) ## What else can you do next? ### Cleaning up and archiving the old backups to Glacier We actually never delete backups. If you do want to do that, you can make duplicity do it - add something like _duplicity remove-older-than 6M --force s3+http://example-bucket/backups_ We prefer to use a lifecycle rule, that migrates backups older than 100 months to glacier (make sure this time is longer than the time between full backups you've given duplicity, otherwise you can't restore anything without digging into glacier). So far we keep them there forever, but we might further expire ### Monitoring your backups I’m a firm believer that you need to monitor your backups automatically. When they die you want to know quickly, not when you next go to restore a backup and they don’t work. One way to do this (certainly not a catch-all) is to at least check that something is being written to your s3 bucket ever 24 hours. For this you need to enable [request metrics in S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html) (which costs extra), then setup an alarm when the sum of those over 24 hours is zero We've create a metrics filter in cloudwatch logs looking for the phrase "error".  The alarm on this sends us an email when there's an error in duplicity. What more could we do? Parsing the output of duplicity collection-status is what we’ve done in the past, using [nagios](https://camille.wordpress.com/2017/09/20/incremental-backups-with-duplicity-plus-nagios-monitoring/). We’ll probably try to hook something up using ECS & Cloudwatch to do this for us. You’ll also want to schedule something in your calendar to remind you to do a test restore every so often. \]\]>https://idealstack.io/blog/codebuild-2-speeding-it-using-caching-adding-code-coverageCodebuild for PHP - Speeding it up using caching, adding code coverage2018-01-15T13:00:00+13:002018-01-15T13:00:00+13:00IdealstackTo continue our series of codebuild posts - how to use Codebuild's caching to speed up builds, and how to generate reports of code coverage using PHP Unit...col-sm-12"> This is part 2 of our series on using AWS Codebuild with PHP projects.   You probably want to start with [part 1](/content/blog/setting-aws-codebuild-test-php-project/index.html). In this article, we'll discuss speeding up our builds with caching, and generating a code coverage report using PHPUnit (you could use the same techniques to upload other code quality metrics from various other PHP code analysis tools too if you want) As with the previous posts, we're using this Github repository [https://github.com/Idealstack/codebuild-example](https://github.com/Idealstack/codebuild-example) \- you can follow along by either making your own fork of this, or copy the relevent _buildspec.yml_ files into your own project and tweak them appropriately ## Speeding up codebuild using caching Codebuild can cache data, which allows us to speed up the build.  Since we're installing a lot of debian packages in our build it helps to cache the apt packages used by ubuntu to install PHP.  Any PHP project can also probably benefit by caching composer packages.  If you are using nodejs based tools like gulp or webpack in your build then caching the packages for that is another win. 1. Create an S3 bucket to store your cache. You can just accept all the defaults when creating the bucket 2. Edit your codebuild project 3. Change the ‘Cache’ setting to use S3 4. Choose your S3 bucket, then Save 5. Use the build spec file codebuild-2-caching.yml 1. 2. The first time you build after this, you’ll see it takes a while But in the build logs it’ll upload cache The next time you build, it will be faster: As you can see, the build takes about a third as long.  And you should see in the logs that it uses cached files in composer, npm and apt-get install commands ## Code coverage reports PHPUnit can generate reports on the coverage of your unit tests.  You probably want this to be generated automatically too, so that when you are code reviewing a commit or pull request you can know whether it has improved or worsened your code coverage. To get an html report on code coverage from your builds, first create a new s3 bucket to hold the report. Then update your build specification to use buildspec-3-coverage.yml. Edit your project, click to show Advanced Settings, and add an environment variable called _COVERAGE\_S3\_BUCKET_ \- set it to the name of the bucket you created above Also take note of the “Role Name”, which you’ll need in the next step Now you need to give codebuild access to this s3 bucket. To do this you need to allow access to the IAM service role that codebuild creates 1. Go to IAM in the AWS “Services” menu 2. Go to “Roles” in the IAM menu 3. Find the role that codebuild is using 4. On the “Permissions” tab click “Add inline policy” 5. Go to the JSON tab and paste the following. Update the bucket name (in this example, codebuild-example-test) ``` { "Version": "2012-10-17", "Statement": [\ {\ "Sid": "CodebuildCoverageAccess",\ "Effect": "Allow",\ "Action": [\ "s3:ListBucket",\ "s3:DeleteObject",\ "s3:PutObject"\ ],\ "Resource": [\ "arn:aws:s3:::codebuild-example/*",\ "arn:aws:s3:::codebuild-example"\ ]\ }\ ] } ``` 6. Click “Review Policy”, Give the policy a name eg “CodebuildCoverageS3Access” & click “Create policy” Now when you rebuild your project you should see the coverage uploading How to view this report? We don’t want to make the coverage report public - there’s potentially all sorts of security-sensitive information in there. The best solution we’ve found so far is to limit it by IP Address 1. Go to the bucket in S3 & go to the permissions tab 2. Click the Bucket policy button 3. Paste in the policy: update the bucket name and IP and Save ``` { "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [\ {\ "Sid": "IPAllow",\ "Effect": "Allow",\ "Principal": "*",\ "Action":["s3:GetObject"],\ "Resource": "arn:aws:s3:::examplebucket/*",\ "Condition": {\ "IpAddress": {"aws:SourceIp": "54.240.143.0/24"}\ }\ }\ ] } ``` Now you should be able to navigate to the index.html in the s3 bucket and click the link And view the coverage report ### What's next? In [part 3](/content/blog/codebuild-deploying-using-sftp-idealstack-or-any-similar-system/index.html) we'll show you how to deploy automatically over ssh \]\]>https://idealstack.io/blog/codebuild-deploying-using-sftp-idealstack-or-any-similar-systemCodebuild for PHP - Deploying using SFTP (to idealstack or any similar system)2018-01-15T13:00:00+13:002018-01-15T13:00:00+13:00IdealstackHow to use codebuild to deploy your code automatically via SSH/SFTP...This article is part of a series on setting up codebuild, so you might also want to check out [part 1](/content/blog/setting-aws-codebuild-test-php-project/index.html) and [part 2](/content/blog/codebuild-2-speeding-it-using-caching-adding-code-coverage/index.html) first. Once you've got Codebuild building and testing your code, the obvious next step is to get it to deploy automatically when it passes tests - this is called Continuous Deployment. Continuous Deployment is an intimidating concept when you first hear about it - surely you want to manually test your releases a little bit first before putting live?!  But you have to ask yourself: how much do you really test each release.  Once you have good unit tests in place, the chances are that you'll just release code as a formality, without any manual testing.  In which case you might as well automate it.  Put all the time you spent on manual testing into making better unit tests.  If particular features need manual testing - make sure that is part of your code review process before you merge the changes. In this article we'll show you how to upload files over SFTP and run some SSH commands to deploy your app - you should be able to easily adapt it to deploy any php app. The challenge to implementing this in codebuild is to safely store your ssh key. You definitely don’t want to commit this into your repository in the buildspec.yml, or just throw in an S3 bucket and hope for the best. A safer approach is to use the AWS Systems Manager Parameter Store and encrypt the key using AWS KMS 01. Go to AWS Systems Manager 02. In the AWS Systems Manager menu, choose _Parameter Store_ 03. Create a parameter 04. Give the parameter a sensible name, description, and choose the type to be _SecureString_ 05. Generate an SSH private key, and paste it into the value field 06. Paste in the SSH private key.  You should generate a unique key for every project, one that only has access to your hosting server - don't just use the same SSH key you use for your everyday use here. 07. Now you need to give codebuild access to this s3 bucket. To do this you need to allow access to the IAM service role that codebuild creates 1. Go to IAM in the AWS “Services” menu 2. Go to “Roles” in the IAM menu 3. Find the role that codebuild is using 4. On the “Permissions” tab click “Add inline policy” 5. Go to the JSON tab and paste the following. Update the name of the key  (in this example, codebuild-example-deploy-key) ``` {     "Version": "2012-10-17",     "Statement": [\         {\             "Sid": "CodebuildAccessDeployKey",\             "Effect": "Allow",\             "Action": "ssm:GetParameters",\             "Resource": "arn:aws:ssm:*:*:parameter/codebuild-example-deploy-key"\         }\     ] } ``` 6. Click “Review Policy”, Give the policy a name eg “CodebuildAccessDeployKey” & click “Create policy” 08. Edit the codebuild project and add environment variables  - SSH\_SERVER, SFTP\_PORT, SSH\_PORT, SFTP\_PORT, SSH\_USERNAME referring to the ports, username and server IP addresses of your SFTP/SSH server.  Then add a variable called SSH\_KEY, choose the type to be _Parameter Store_ with the name of the encrypted parameter you created above 09. Add this code to the post\_build section of your buildspec.yml. ``` # Deployment to the SFTP server - bash ./codebuild-deploy.sh ``` 10. The real work happens in [_codebuild-deploy.sh_](https://github.com/Idealstack/codebuild-example/blob/master/codebuild-deploy.sh) \- You'll want to tweak the directories copied and the remote commands needed  depending on your own project. ### What could you tweak or change here? We've tried to keep this example generic, but in real world projects there's a couple of things you might want to do differently 1. Consider using [Deployer](https://deployer.org/) for more sophisticated  zero-downtime deployments. 2. If instead of deploying over SSH/SFTP you've got more complex deployment requirements on EC2/ECS etc, AWS Code Pipeline is the right AWS tool for the job.  You'll end out using the Codebuild setup we've given you but without this deployment step \]\]>https://idealstack.io/blog/setting-aws-codebuild-test-php-projectSetting up AWS codebuild to test a PHP project2018-01-14T13:00:00+13:002018-01-14T13:00:00+13:00IdealstackAWS Codebuild provides a reliable, cheap & easy way to automatically run your unit tests with every commit. We explain how to get CodeBuild working well for PHP projects....AWS Codebuild provides a reliable, cheap & easy way to automatically run your unit tests with every commit.   We use it here at Idealstack to test and deploy. Like a lot of AWS tools though it doesn't really support PHP "out of the box" and you've got to mess around to get it working well for PHP projects.  Thankfully that's not too hard, and since we've gone through the work to get it running we thought we'd share it for your benefit. ## Why use codebuild? You need your tests to run every time you commit a new piece of code to github.  If you run your tests manually they’ll get forgotten. I firmly believe there’s no point in writing unit tests unless you also automate them. Codebuild (like most build tools) can report your build status to github, which gives you a handy indication of whether your builds are passing or failing for each commit, branch & pull request If you're on AWS, I think the best way to get automated testing happening is AWS codebuild. It’s cheap - for many projects free under AWS's free tier (you get 100 free build minutes per month).  You only pay for the builds you use, there's no fixed monthly fee, which makes it great for projects in maintenance mode that don't need of builds.   It’s simple to setup and seldom needs ‘fixing’.  And like most AWS services it does just one job and does it well, but you can hook it together into all the other AWS services to create any kind of custom solution you want. The only problem is that, like a lot of AWS services, PHP is not a ‘first class’ supported language on Codebuild and you have to jump through a few hoops to get it going. ## Let’s get started - setup codebuild to build your project Firstly, let’s create a simple PHP project. We’ve used the Lumen microframework here, but really, any PHP code is fine: [https://github.com/Idealstack/codebuild-example](https://github.com/Idealstack/codebuild-example) If you want to follow along with this howto you might want to fork this repo on github.  Or you might prefer to cut out the double-handling and setup codebuild on one of your real projects using these instructions. ### Setup a new codebuild project 1. In the AWS console, go to CodeBuild 2. Click "Get Started" 3. In the section _What to build?_  Find your repository on githib. We recomend you check the option  _Webhook_ (so builds happen automatically on commit) and the build badge (so codebuild updates pull requests, branches in github with build status). 4. _How to build:_ Choose an ubuntu image. For the runtime we’ve chose Node.js, since we might be using node tools such as webpack in our build. If you don’t need this just choose ‘Base.’.  Choose to get your buildspec from the source code root directory, and leave the buildspec name on the default of buildspec.yml 5. Choose to do nothing with the artifacts for now.  You might want to do something such as upload them to s3 once you've got this working 6. Leave all the other settings on their defaults (we’ll talk about what you might use them for in a moment) The most important thing here is the _codebuild.yml_ file. This is a file you should create in the root of your repository that tells codebuild what to do. In our case, we need to bootstrap a PHP environment and run our unit tests. There’s three ways to do that : one is to create a docker image, push it to ECR, and use that. But it's hard to keep this image up to date, especially if you aren't already using docker elsewhere. Or you can use a pre-existing docker container - if you strike it lucky and find one that’s well maintained configured exactly how you want it, that’s a good option. Assuming you're not already using docker though, and you don’t want to dig around finding a docker image that suits you, don’t panic. You can pretty easily just use the default Ubuntu image codebuild provides - just install what you need as part of the build process. The buildspec.yml included in this repository does this : [https://raw.githubusercontent.com/Idealstack/codebuild-example/master/buildspec.yml](https://raw.githubusercontent.com/Idealstack/codebuild-example/master/buildspec.yml) It essentially just uses an inline script to install PHP, mysql or whatever else you need onto the stock ubuntu image that codebuild provides. In this example we've installed php7.1 (you should just be able to change the version number to whatever you want), mysql (you may not need this - Laravel, Lumen etc often just use sqllite when running unit tests, although sometimes your code needs real Mysql to run. Here at Idealstack we push Mysql hard enough that SqlLite just won't work for us.   We also install gulp, grunt & webpack (remove these if you don't use them) See [buildspec.yml on github](https://github.com/Idealstack/codebuild-example/blob/master/buildspec.yml) Assuming you've got this buildspec file in place, you should be ready to build your project (either by doing a commit or clicking the "Start Build" button in Codebuild. With luck, you should see your tests pass: ## Setup notifications for codebuild The other thing your most likely going to want is an email to tell you when your build fails.  You can acheive this by setting an event handler in Cloudwatch 01. Firstly, we setup an SNS topic to notify us.   Go to the _Simple Notification Service_ console in AWS 02. Create a new topic 03. Check the topic and choose Subscribe to Topic 04. One the Create Subscription screen, choose the protocol as Email and enter your email address as the endpoint: 05. Go to the Cloudwatch console in AWS 06. Under _Events_ in the Cloudwatch menu, click _Create Rule_ 07. Configure the Event Source like so: choose "CodeBuild" as the service name, "CodeBuild Build State Change" and choose the states you want to be emailed for: 08. Under targets: Configure an SNS Topic.  Select the SNS topic you created 09. In the _Configure Input field_, choose "Input Transformer".  In the Input Path paste this: ``` {"build-status":"$.detail.build-status","project-name":"$.detail.project-name","build-id":"$.detail.build-id","region":"$.region"} ``` Then in the "Input Template" paste this: ``` "'' has build status of ''. https://us-west-2.console.aws.amazon.com/codebuild/home?region=#/projects//view" ``` 10. Now (once you confirm your email address for the SNS subscription) you should start receiving build notification emails when your builds complete. ### Credits When we were originally setting up codebuild, we relied on [Ben Ramsey's helpful post](https://benramsey.com/blog/2016/12/aws-codebuild-php/).  Ben does it a bit differently, you might want to check out his approach as well. ### What's next? In the [next post](/content/blog/codebuild-2-speeding-it-using-caching-adding-code-coverage/index.html), we'll show you how to cache the build so we don't need to keep downloading composer and ubuntu packages on every build.  This will speed things up a bit.  We'll also upload code coverage reports from PHPUnit.  Finally in [part 3](/content/blog/setting-aws-codebuild-test-php-project/index.html) we'll show how to deploy automatically over ssh \]\]>