Security management has become a vital part of a website administrator’s role. With hacker attacks taking place every few minutes, Drupal administrators are finding there’s more to do when it comes to securing the site, troubleshooting, and resolving problems caused by malicious code.

This blog fleshes out the published recommendations from Drupal about what to do after an attack. Remember that your website depends on more than a Drupal database and files: it’s also part of a webserver environment so you need to look at system components other than Drupal. This blog is useful reading for both managers and Drupal admins because many decisions are not just technical ones. You must also take business risk into consideration.

1. Take notes

This is a stressful time. Don’t rely on your memory. Be methodical. Use Notepad or some other tool to document:

• each problem you discover
• the steps you take to remedy the problem
• what you can do to prevent it in the future or at least to investigate how to protect against this problem in the future

2. Make forensic copies of the site

Make a copy of the Drupal database, files, and your operating system environment before you do anything. Keep a copy on external storage medium (DVD, SD, RAID, etc.) and store it offsite. 

You can also make backups along the way as you clean up your files. If you do, put them in a different directory than for your regular backup files. These ‘working backups’ are useful in case:

• any changes you make to fix the problem end up creating more problems; you can roll back to a previous version
• you need to bring in a third party to help you troubleshoot; it provides a complete picture from before you started to do your own troubleshooting
• if you find something interesting, you can send it to Drupal security for their information and help the community.

3. Scan your servers and PC

Your first instinct may be to change your passwords to prevent hackers from logging in, but before you do that, scan for malware and viruses because:

• The attack could the kind that notifies the hacker of your password changes – which still leaves you vulnerable.
• The problem could be malware or a virus put there by the hacker or introduced by one of your own users via an infected PC.

There’s no sense in uploading a clean backup of your Drupal files and database to an infected server or via an infected PC. Malwarebytes is a popular and cost-effective anti-malware product for Windows PCs, Macs, and Windows servers. For viruses, AVG for PCs and Macs is free and they also have versions for Windows servers. Unix/Linux operating systems don’t get infected as much, but it happens. BitDefender gets high ratings from the AV Comparatives independent test site for anti-virus software, as does ESET. Here’s some good reading about antivirus threats for Linux.

Then, make a working backup of your scanned server(s) files.

4. Check for backdoors

Hackers are getting smarter. Not only do they get into your system, they can edit files or deploy tools to get back in after you’ve performed a cleanup and restore. Search for those backdoors.

• Check passwords: starting with the most privileged accounts: admin passwords, control panel passwords, FTP passwords, etc.
• Check your access logs: you may need a developer or someone familiar with PHP to assist. Look for POST requests and PHP scripts added to directories with writable access.
• Check email rules and filters: are messages being forwarded to unauthorized email addresses? Have answers to security questions changed? How about the security questions themselves?
• eCommerce: many hacking attacks are for mischief only, but others try to get money. If you have any sort of eCommerce set up, do a thorough investigation to make sure all shipping addresses, payment methods, linked accounts, and credit card addresses are legitimate. Worst case, hackers can link a debit account to your account and request money transfers from yours or a client’s credit card. If you’ve reason to believe an account has been compromised, reset the passwords for all services related to that account. An email account is often just the first step for getting to online banking. If you suspect serious incursions, you may have to freeze your credit card or banking account until the issue is solved. If you’re not sure how to investigate, get help for a security audit of your account(s).  

5. Notify stakeholders

If your website has users and you believe the attack has made their email addresses, user profiles, or other personal information vulnerable, you’re morally and sometimes legally obliged to let them know. First, you should inform your organization’s leadership team so that there’s agreement on how to proceed with user and/or client communications. Hopefully you already have IT policies in place about informing affected users, especially if they are clients.

6. Take your website offline

In many cases it’s not possible to take down your website right away; however, if your site has eCommerce, you should take it offline ASAP.

Once you’ve had a chance to do some assessment of damage and risk, you need to decide whether it’s necessary to take your website offline – along with any other servers, including the internal network. Damage isolation prevents follow up exploits and further infections while you clean up and restore. Note: It also tips off hackers that you’ve discovered the incursion and are taking remedial steps.

Ultimately, the decision to disconnect from the Internet is one of risk tolerance. Do you disconnect one system, some of the systems, or the entire organization? Taking down one server to fix it leaves open the possibility that if other servers are compromised, they’ll re-infect the clean one. You may never get to fully evaluate the extent of the attack. The issue becomes one of risk tolerance: for the sake of uptime, how much risk is your business willing to accept?  These days, the desire for security generally outweighs the need to for systems availability.

Read this article on how to use a 503 status code to let search engines know that your site is temporarily offline. Otherwise you might get penalized and lose the SEO advantages you’ve gained over the years. With a 503 status code, Google will not punish your site for being down.

Note: the 503 error code page should be on a totally different server than your website. If you want a more ‘graceful’ message, you’re free to customize the print statement and you can even include a URL that visitors can click to get to a static informational page on a totally different server than your website. Search engines won’t read past the header information.

By the way, putting Drupal into maintenance mode is not the same as taking the website offline. If your site has been hacked, it may have been done in a way that allows the hacker to get in as long as the site is online.

7. Inform your hosting service

If you have a hosting provider, definitely let them know about the breach and that you've taken your site offline. This allows them to:

• inspect their own systems for incursions
• be on alert in case you need their assistance

8. Take extra precautions for eCommerce and client information

You have some extra work ahead if your website:

• manages payment process on the site (i.e. not using/forwarding to a 3rd party provider such as PayPal),
• stores any client data on the web host or uses Drupal to process it, or
• uses any data POST method to send form data via e-mail

If there has been a breach, here are a couple of initial steps to validate users and prevent private data from being easily available. Depending on your situation, you may want to bring in developer or external resources for more thorough investigation.

• Check logfiles to verify that hosted client information has not been altered, downloaded, or copied.
• Update your SSL certificate and confirm it's still secure.
• Implement AVS (address verification system) if your e-commerce system doesn't have it already; at a minimum, add CVV (card verification value).
• Encrypt connections using SSL/TLS to back-end services such as e-mail servers that are being used to send confidential information.

• For a good high level explanation of SSL read How SSL and TLS works. For more technical recommendations, read the white paper Drupal PCI Compliance.

9. Investigate the source of the exploit

(Yes, even if you’ve done Step 8 for eCommerce, it’s not over)

There’s always pressure to get the website back up as quickly as possible. If Drupal has released a security update and your problems can be tracked down 100% to that issue, the odds are pretty good that you can just restore from a backup, apply the Drupal security update, and get back online.

Unfortunately, nothing is ever 100%. Consider the following situations:

• The malicious code could’ve been triggered weeks or months after it entered your Drupal system - in which case, if you don’t know what to look for and clean out, your website will be re-infected after another triggering event.
• The malicious code could exist outside of Drupal – in which case, your website will be re-infected even if your Drupal backup is pristine.

That's why it’s important to understand the source and extent of the exploit. Before you bring a backup of your website online, audit the backup on a staging server to correct/remove malicious code, files, and unauthorized settings.

Investigation is largely a matter of looking for file discrepancies before and after the attack. That’s why recovery is 900% easier if you back up regularly than if you don’t. For tips on backing up Drupal, read last week’s blog about the Backup and Migrate utility.

• If you have a clean backup you’re in a good position. You can use tools to compare files (before and after the incursion) and quickly learn which ones have changed. You can use Kaleidescope, Sourcetree, or just the git command line to spot differences.
• If you don’t have a clean recent backup, you will need to review files and databases manually for suspicious activity.
• When you inspect suspicious files:
  • The end of the file is a good place to start. Look for unusual Javascript or an iframe. Save this code in an external file for future reference
  • Check for links to external URLs or externally hosted Javascript.

The following is by no means a comprehensive list of places to look for suspicious activity:

• .php files, .html files – check for modifications to code files. index.php and .htaccess are the usual suspects.
• Newly modified or created files – a good trick is to sort files by time last accessed and time created. If a hacker has messed with a file, or added a file, it will have a recent timestamp. Go beyond code/executable files (html, js, php) and check for image files (gif, jpg, png) and compressed files (zip, tz).
• In writable directories and database – search for executable code in blocks, nodes, fields, even user profiles. Links, iframes, blog posts, source files.   
• New or modified user accounts – do the email addresses seem logical? Check with the person.
• Sessions table – are there open sessions for users you didn’t create, or who you know have not logged in recently? Look for users with privileged roles and make sure the IP address for the session makes sense.

If you have access to technical or developer resources, here is an article with more comprehensive information about identifying malicious PHP files.

10. Restore your website

Another decision to make after assessing damage and risk is how you’re going to repair your site. Before you make any changes, make a backup if you haven’t already done so.  Don’t work directly on your production site -- recover and/or make your fixes on a staging site. After you’ve made your fixes, remember to apply the latest Drupal security updates.

Option 1: Restore from backup

The easiest and quickest route. If you’re confident you know the date and time of the attack and you’re fairly certain nothing outside of Drupal was affected, you’re in good shape. Just restore the last version of your Drupal website from before the attack. If you’ve been using a utility module such as Backup and Migrate, this is very simple to do.  

Option 2: Rebuild

The cautious route. This is an option for those times when: you don’t have a clean, recent backup; you’re not sure when the attack took place; or you believe damages are too widespread to troubleshoot and fix in a timely manner. Also, this can be a good option if you were already planning to rebuild the site.

This involves downloading and installing Drupal core and contributed modules from scratch. Then you need to add in custom modules, custom themes, configure your settings and users, and migrate content. The amount of work for this option depends on how much site-specific information you’ve backed up safely or can bring over from some other electronic format.

• Custom theme: if you developed your website in-house, hopefully there’s a backup of your custom theme from your development project. If an outside agency built your website, they may have a copy.
• Configuration of settings and users
• Content and images: check with Marketing and content editors. Word documents? Archived images?

Option 3: Keep and fix manually

The investigative and forensic route. This is something to undertake only if you have in-house and/or external technical resources. If you really don’t have any useful backups of anything, this could be your only option. If you backup regularly but want a thorough analysis of the attack, the information you gather during this process will assist you in validating and developing tighter security.

11. Check your recovered site before going live

Always worth another reminder: audit your recovered site to correct/remove any malicious code, files, or settings. Close up those backdoors.

12. Revisit your decisions

Keep your business goals in mind. If at any time during this recovery process you find evidence that the attack was worse than or (hopefully) not as bad as you first thought, you may want to change your plans for recovering your website and webserver environment.

Conclusion

Many of the decisions you make must be guided by your business as well as technical objectives. The best way to reduce the risk and effort of recovering from a hacker attack is to be prepared. Regular backups and documented procedures help prevent going into crisis mode. If you lack resources to ensure regular updates and backups for your website, contact us for more information about Smartt’s web services.

Useful links for staying on top of Drupal security issues:

Check out the resources on the Drupal Security Advisories page for more information and links:

• Security advisories news page
• Security public service announcements
• RSS feeds for core, contrib, or public service announcements 
• Drupal security on Twitter
• Subscribe to email list for security newsletters

Remember that if you uncover a Drupal security issue, do not post about it on any forums. Keep it confidential by submitting it to the Drupal Security Team.