This outage list was specifically created for you on 2024/04/26 at 6:07:05 PM

Colour Legend

Ongoing

No ongoing outages or bulletin notices at this time

Resolved / Archived

MySQL is unavailable
Impact start time: February 1st 2021 at 7:52PM Eastern
Resolution time: February 1st 2021 at 10:42PM Eastern
Status: Resolved

UPDATE: This has been resolved. All databases are now available.

MySQL is currently unavailable. We are investigating the cause for the outage and are working on restoration now.

Ticket system is offline
Impact time: January 15th 2021 at 9:25PM Eastern
Resolution time: January 16th 2021 at 1:40AM Eastern
Status: Resolved

Update (Jan 16 2021 - 1:40AM Eastern): This maintenance has concluded and the ticket system is fully available.

The ticket system (https://exalthost.com/tickets) is currently undergoing unscheduled maintenance. We anticipate restoration later tonight.

If you need support or assistance during this time: please email support@exalthost.com. Existing tickets will not be impacted and will be answered as soon as possible.

The ticket system is currently asking for a username/password during this time. There are no valid credentials for this prompt. This is intentional while we complete this unscheduled maintenance.

Thanks for your patience! We will update this page as work continues.

UPDATE: COMPLETE | Planned maintenance to improve network performance on serv03-d01
Impact time: October 21st 2020 afternoon
Status: Maintenance Completed

UPDATE: This was completed without incident. Results should be noticeable

We are performing maintenance that will result in server access becoming unavailable in the afternoon for approximately 5-10 minutes.

This maintenance will be improving network performance for some servers. This improvement will be exponential and noticeable for most customers. We appreciate your patience as we upgrade our services to serve you better!

PHP down
Impact time: June 22nd 2020 at 12:13AM Eastern
Status: Resolved

On June 22nd 2020 at 12:13AM, we were informed that the PHP service was no longer running. Our automated recovery script failed to start PHP. We began working on the issue immediately and determined that a package upgrade had moved a required binary and was causing PHP to fail to start.

We made adjustments to our PHP installation to resolve this issue and all PHP services have been restored.

Moving forward, we will run a new test to ensure this does not happen again.

May 31st 2020 - System upgrade complete
Impact time: 3 hours starting May 31st 2020 at 5PM Eastern
Status: Completed

On May 31st 2020, we completed planned configuration changes to improve our service. Part of these changes were improvements to our kernel patching process, which will greatly reduce the frequency of server downtime (due to reboots). Although the downtime was always minimal, it could've interrupted ongoing transfers or other processing that may have been occurring when we decided we needed to restart the server.

By reducing (and effectively eliminating, for the most part) unexpected server restarts, we've created a more predictable and stable server experience. This means that there is no more "risky/maintenance" periods in the early morning (eastern) where the server may terminate connections for a quick restart.

Hope this helps! If you have any questions, please create a ticket.

October 22nd 2019 - Backup system temporarily impacted
Impact time: October 22nd at 3AM Eastern
Status: Resolved
Cause: Backup server outage and software glitch
Scope: Impacted one nights backup. Did not affect previous successful backups
Resolution: Restored backup server and reconfigured backup software

Update: The backup system is now confirmed as functioning normally
Our nightly backup failed to properly execute, resulting in no backups being made on October 22nd 2019. This issue was noticed by administrators upon daily backup verification routines. This issue was caused by two separate, but connected issues:
1) The proprietary backup software we've written on the newly allocated servers was missing some logic that could result in failures to move backups to a secure off-site location
2) The remote backup systems route failed and couldn't be reached from any of our datacenters.

Both of these issues have been resolved and we expect nightly backups to work as intended, and will update this post as the backup system attempts its next nightly backup.

This issue did not impact previous backups that were successful in the hours before this attempted backup; however, it did mean that no backup was made on October 22nd 2019. Once the backup software was improved, and the secure remote route became active once again, a manual backup took place which was successful.

The steps taken in solving issue #1 will make sure this issue will not happen again.

SSL certificate mismatch for connections without an SNI request
Impact time: April 28th 2019 - night

Late on April 28th 2019 we became aware of an SSL certificate issue for connections without an SNI request. We have always recommended the top level domain of exalthost.com for connection types without SNI support, or for clients that do not support sending an SNI. This is because for connection requests that do not provide an SNI, we serve the certificate for exalthost.com whenever possible (except for some servers, but those users are aware). On April 28th, servers began serving a valid certificate for the hostname of the server for which your account lives on. This would create a mismatch error if you used exalthost.com as your connecting hostname.

We corrected this issue very quickly and are now consistently returning the proper certificate for exalthost.com, regardless of where your account lives.

This was not a security issue, and could've been temporarily solved by using the appropriate hostname on your connection request until we reset the certificate.

Server Upgrade/Maintenance Completed
Impact time: December 3rd 2018 at 1am Eastern to December 3rd 2018 at 4am

We apologize for an outage that lasted from 1am Eastern on December 3rd 2018 until ~3am Eastern on the same day. This outage was necessary to transfer data to a new server installation with upgraded software. Please verify all files to ensure everything transferred properly. To protect your privacy, we do not inspect transferred data and will require you to review all your data. We will hold onto the old server data for 3 weeks from the time of this post before securely deleting it.

We have successfully completed our planned server upgrade from CentOS 6.10 to CentOS 7. This was in response to CentOS's deprecation plan for CentOS 6 (which was to be deprecated in 2020). We are now running a more modern version of the operating system.

All accounts were transferred to the new server without any issue and you don't need to do anything on your end. If you notice any account issues after the transfer, please contact support at support@exalthost.com or by creating a ticket. The old server will be decommissioned in 3 weeks once we confirm all users are satisfied with the transfer.

Some details:
- The transfer was completed without any major incidents
- IP addresses have been re-assigned, but some sites may need a manual DNS zone update. If you're having issues, please let support know. Preliminary checks have found no issues with IPv6 or IPv4.
- SSL certificates were transferred from the old server. It does not require re-installation
- DNSSEC has been disabled. If you have enabled DNSSEC, you'll need to remove the old DNSSEC settings at the registrar level, and create a new entry in cPanel and set it back up. Impacted customers have been notified.
- If you had custom exceptions from the support team for PHP (for example, shell access), you'll need to reach out to support to get these enabled on affected domains
- WebDAV is still under maintenance and may not be reliable at this time. We will notify customers when WebDAV maintenance is complete. For now, we recommend SFTP, FTPS, or SCP
- Cleartext FTP has been re-enabled after the transfer. We've temporarily re-enabled cleartext/unencrypted FTP on port 21 while we complete maintenance on some service certificates. Do not rely on cleartext FTP as we plan to disable it again in early 2019.

This post is still being updated.

OS Upgrade Plan Announcement
Last update: October 31st 2018

As many of you know, CentOS has announced EOL for CentOS 6. All of our servers currently run the latest version of CentOS 6 (6.10 as of writing). To make sure we are prepared for the future, we are upgrading to CentOS 7. The upgrade path is still being identified and planned, and we will announce it ASAP.

We will formally announce our migration and upgrade plan shortly. Please stay tuned. We aim to minimize downtime as best as we can.

March 8th 2016 - Network outage
Time to repair: Solved within 2 hours
Status: Solved
Scope/Issue: Affected all sites on the giant server
Solution: Network provider issue. Network provider solved

Description:
On March 8th 2016, our IPv4 network path was made temporarily unavailable. Some users may have been unable to access your website depending on their ISP's routing. This issue was with our network uplink and not our server. The service was restored within 2 hours, and we are monitoring the route to watch for potential future issues to catch it faster should it occur again.

February 2nd 2016 - Site outage
Time to repair: Solved within 10 minutes
Status: Solved
Scope/Issue: Affected all sites on the giant server
Solution: Revert to previous software and re-build Apache

Description:
On February 2nd 2016 at about 3:00PM EST, all websites went down. Your users would have seen a '403 Forbidden error' along with a message similar to 'unable to read htaccess.' This issue was due to a failed upgrade to the new PHP/Apache management system on the server. We resolved the issue by restoring the previous management software temporarily. All websites were made available within 10 minutes of the outage occurring.

The new PHP/Apache management software will allow us to make multiple PHP versions available to you, and offer almost every available PHP option and extension without you needing to contact support. You'll be able to control the PHP version in your cPanel. We hope to launch this service as soon as all issues discovered today are resolved.

August 17th 2015 - Severe slowdowns with some ISPs
Time to repair: Solved
Status: Solved
Scope/Issue: ISP's routing through Cogentco
Solution: Worked with ISP's to determine solution.

Description:
Update (August 18th 2015): We are now routing through Telia while we identify the problem.  Speeds should be back to (or very close to) normal.


Some ISPs are experiencing severe download slowdowns.  This affects any ISPs where the return route to ExaltHost servers involve Cogentco, and does not only affect Cogentco (e.g. Imgur, Vimeo, and other sites are also affected). Details on this issue can be found at http://www.dslreports.com/forum/r30224258-rCable-imgur-com-slow~start=30.

The affected ISPs (Start, TekSavvy, Rogers, and potentially many others) are currently working with us to determine the routing bottleneck and rectify the issue. As of right now, using IPv6 to access your site is the best workaround until we identify the issue.

June 2015 - IPv6 now available and assigned to all sites

Description:
We are very proud to announce the release and activation of our IPv6 assignment.  All users have an IPv6 address assigned and nameservers are now responding to v6 requests. It's important to make sure to test your PHP scripts (and any other scripts for that matter) to ensure that they properly recognize IPv6 visitors.

Users on IPv6 and IPv4 dual-stack will be presented with the fastest response.  If IPv4 answers first, your visitors will visit your site through IPv4.  If IPv6 responds first, your visitors will visit your site through IPv6.

All websites have a dedicated IPv6 address assigned to their domain regardless of whether or not they have purchased a dedicated IPv4 address.  This IPv6 address is assigned free of charge; however, you will still be using a shared IPv4 address unless you've purchased a v4 address. You can find your IPv4 and IPv6 address assignments in cPanel under "Server Information" (on the sidebar in the left of the panel).

Reminder: IPv4 addresses are available for lease at $2.00 per month by opening a ticket on our main site. 

October 26th 2014 - SSL POODLEbleed vulnerability... SSL3 disabled
Time to repair: Fixed
Scope/Issue: A vulnerability existed within SSL 3.0 installed on our servers
Solution: Disabled SSL 3

Description:
The SSL 3.0 vulnerability proves to be present even with reconfigured ciphers. As a result, we've disabled SSL 3 on our servers. Many companies chose not to do this, for compatibility reasons. We have chosen security over backwards compatibility.

As a result, IE 6.0 and older versions of Java will no longer securely access our server, and therefore can no longer login to cPanel, or webmail. We understand that this is frustrating, but security vulnerabilities abound with older IE versions.

October 17th 2014 - SSL POODLEbleed vulnerability
Time to repair: Fixed
Scope/Issue: A vulnerability existed within SSL 3.0 installed on our servers
Solution: Updated OpenSSL and reconfigured cipher

Description:
An SSL 3.0 vulnerability was found titled POODLEbleed.  We have updated all servers and performed configuration changes to the cipher. No breach or security threats are currently reported to have occurred.

If you have an SSL certificate installed, there is no reason to issue a new one.  This vulnerability exists with the SSL server and not the certificate itself. Your website has been secured as well as all cPanel urls.

Decommissioning of kingvarian and addition of new server - Status updates
Last update: September 28th 2014 at 2:21AM EST
Transfer COMPLETE!  Welcome to your new server.  Please check all data (including dynamic, MySQL, email, static, etc) and contact support for assistance if needed.  Enjoy your greased up, blazing fast new server.

Previous updates
September 27th at 01:05PM: Two new servers ready for transfer.  Transfer will begin and complete within 14 hours.
September 23rd at 01:49PM: Transfer delayed due to DNS issues at the registrar level. 
September 23rd at 12:00PM: DNS update submitted.  Transfer will begin upon successful nameserver update. ETA 2 hours
September 23rd at 11:00AM: Plans for transfer made publicly available
September 23rd at 10:50AM: New server is fully configured and sitting idle on the internet. Benchmarks are impressive. Continuing tweaks while kingvarian is prepared for wipe.
NA-MONTH1 NAth at NA:NAPM: New server is being configured and secured.  Inspection will take place as well. ETA 24 hours
September 15th at 03:02AM: Backup of kingvarian completed.

Important Information
We are decommissioning EOL server kingvarian in order to keep up with your sites demands.  In its place, we will be configuring hardware with more RAM, more CPU's and a BGP connection direct to major peers/ISPs.

June 25th 2014 00:30 - Not found errors
Time to repair: Less than 20 minutes
Scope/Issue: All users sites displaying not found errors even if the file exists
Cause: Apache jail broke (possibly relating to failure to chkdir and read htaccess)
Solution: Reset all Apache Jail shell (temporary)

Description:
At about 00:30 on June 25th 2014, we performed a server reboot due to an updated kernel (after 70+ days of uptime) and it caused a malfunction with the Apache jail.  We fixed the issue by resetting all Apache Jailed users.  Some users may require SSH to be re-enabled after this reset.  For the duration of the issue, your users would have seen a Not Found or 404 error when visiting any page on your site, including files that are actually present.  This is not a security issue.

April 11th 2014 - Maintenance
Time to repair: Apr 11 2014 10PM - Apr 12 2014 4AM
Scope/Issue: Maintenance will be performed on the server and network. Packet loss and slowdowns expected
Cause: Scheduled Maintenance
Solution: Scheduled Maintenance

Description:
Our server will be having maintenance performed on it starting on April 11th 2014 at 10PM and ending at approximately 4AM on April 12th 2014. We do not expect any major downtime during the maintenance. However, periods of downtime are not to be completely unexpected. Any downtime is expected to last no more than 15 minutes. We do believe that periods of packet loss and slow loading times (or complete loading failures) may be experienced. We will post more details on this page as the maintenance approaches.

April 9th 2014 - All server accessibility blocked / Heartbleed vulnerability check
Time to repair: Less than 15 minutes
Scope/Issue: The server may have been vulnerable to the recent Heartbleed exploit. We turned off all server access
Cause: Manual server block (internal / staff access only), OpenSSL vulnerability
Solution: Already patched on 0 day. New SSL certificates and keys installed

Description:
The recent Heartbleed vulnerability that has taken the web by storm has caused this outage. We determined that our servers may have been vulnerable. As a result, for everyones security, we disabled all outside access until we could update OpenSSL.

UPDATE: The server was updated on 0 day (at around 7PM). For extra precaution, we have installed a new SSL certificate and new keys. Please empty your cache.

March 15th 2014 - Backup maintenance
Time to repair: All day
Scope/Issue: The server will not be performing or storing backups today
Cause: Backup system failure.
Solution: cPanel technicians will be working on the server

Description:
Our backup service is currently experiencing some unexpected issues. On March 4th 2014, cPanel technicians worked diligently to find a solution to the issue. The issue is still not solved. On March 15th 2014, the server may be slow or show unusual behaviour, especially relating to cPanel. The technicians will be on the server at about 2AM. We will update this page when more information is available.

March 10th 2014 - Access to server blocked
Time to repair: 10 minutes
Scope/Issue: Access to all server resources was blocked
Cause: Examining unexpected file tamper
Solution: Cleared and access granted

Description:
On March 10th 2014, our staff became aware of unexpected file tamper (2 files). Within minutes, it became clear that this was not the cause of any work done by ExaltHost. To protect our users, we disabled all outside access to the server while we looked into these changed files and determined the reason for their change.

Within 10 minutes, research concluded that the files changed due to an OVZcP Script and was no cause for concern. We allowed outside access after this point.

March 4th 2014 - MySQL outage
Time to repair: 2 hours
Scope/Issue: All MySQL databases down for 1 hour
Cause: InnoDB became corrupt
Solution: Disable native AIO and force recovery of all InnoDB databases. Updated MySQL to 5.6.

Description:
On March 4th 2014, the InnoDB storage engine on the server became corrupt. MySQL databases utilizing InnoDB became unavailable. Once we were aware of the issue, we began work on repairs. When repairs began, MySQL went offline due to a side effect from the troubleshooting. MySQL remained unavailable for roughly an hour. To repair the issue, we disabled native AIO and forced recovery of all InnoDB databases.

January 2014 Outage
On January 16th 2014 at approximately 12:00AM our servers suffered an issue which caused all websites and services to become unavailable. Our monitoring brought this to our attention shortly after the outage occurred and work on restoration began immediately.

Service was restored at N/A. The cause for the outage was determined to be N/A. 

No user information was lost during this time and our servers were not breached. We continue to monitor logs for signs of any potential issues/access. There is no need to change anything with your account. 

As part of our apology for this outage, we will be refunding this months fee and providing 30 days of free service. If this is not automatically added to your invoice, please go to our main site and leave a message with a consultant.
Thank you for your patience. ExaltHost Team