Monitor

Following a server restart Hi-mail is back up and running and all services have resumed.

The Hi-mail login page has become unresponsive. Our IT team are looking at it now and we’ll post an updated as soon as we know more.
UPDATE: A server restart is underway to try and resolve the issue.

Hi-mail is now back up and running.

We have been made aware that Hi-mail is running quite slowly this morning and our IT team are looking into the issue now.

Following a server restart Hi-mail is back up and running and all services have resumed.

We have been made aware that Hi-mail is running quite slowly and our IT team are looking into the issue now.

The effected servers have now been restarted and Hi-mail is back up and running as normal.

One of the Hi-mail servers has become unresponsive and requires a re-start. The IT team have been advised and are working on a fix now.

All Hi-mail systems are now back up and running.

Hi-mail servers are still unresponsive following a restart. The IT team have been advised and are working on a fix now.

The issue has been resolved and Hi-mail is back up and running.

Hi-Mail is currently not loading the login screen following a server patch early this morning. IT support are looking into the issue now.

Hi-mail operating as usual following the completion of a Server backup and restart.

We have been made aware that Hi-mail is running quite slowly and our IT team are looking into the issue now.

The server issues have been resolved and Hi-mail is now back up and running.

One of the Hi-mail servers has failed to restart following a scheduled software update. Engineers are looking into the problem now.

Hi-mail servers have been restored and the service is now up and running again.

The Hi-mail login page is not loading correctly following a server update overnight. Engineers are looking into the issue now.

The network connection has been restored and all services are now back up and running.

We are aware of a network issue thats caused Hi-mail to stop responding. IT staff are looking into it now.

Hi-mail servers have been rebooted and the service is now back up and running

One of the Hi-mail servers has become unresponsive and requires a re-start. The IT team have been advised and are working on a fix now.

All services are now back up and running following a server restart.

We are currently experiencing internet issues and BT are looking into this.

The main fibre line has been restored and all services are now back to full capacity and working as expected.

We are having some issues with our BT Fibre line this morning which is being looked into by BT Engineers. Hi-mail is still loading using the ADSL Broadband line but may be slower than normal.

Hi-mail servers and now back up and running.

Hi-mail is not loading – Server Error. The IT team have been advised and are working on a fix now.

Hi-mail is now back up and running following a server restart

Hi-mail has stopped responding. Our IT team are looking into this and we aim to bring Hi-mail back up as soon as possible

Hi-mail servers are now back up and running as normal

Hi-mail has stopped responding. Our IT team are looking into this and we aim to bring Hi-mail back up as soon as possible

The Hi-mail services stopped responding during todays batching process and has since been restarted. All services are now back to normal.

Hi-mail appears to be unresponsive and our IT team are looking into this at the moment. We hope to have the issues resolved shortly.

All Hi-mail services are back up and running once again.

Hi-mail Server has stopped responding. Our IT team are working to bring all services back up as soon as possible.

Hi-mail is now back up and running again following a server restart.

Hi-mail is currently down. Our IT team are looking into this and aim to bring Hi-mail back up as soon as possible

The server issues have been resolved and Hi-mail is now back up and running.

We’re aware of a problem with one of our Hi-mail servers this morning. Our IT team are currently looking into this

The Hi-mail service has been re-started and is now back up and running.

Part of the Hi-mail service has stopped responding and needs to be re-started. Our IT team are looking into this now and hope to have the service back up and running shortly.

The Hi-mail service has been re-started and is now back up and running.

Part of the Hi-mail service has stopped responding and needs to be re-started. Our IT team are looking into this now and hope to have the service back up and running shortly.

All Hi-mail servers are now back up and running again.

We are aware of several servers not responding this morning which are being looked into by IT staff.

Hi-mail is now back up and running although it is running slower than normal. We are still working on this issue and may need to re-start the servers. We will update this entry again if that is the case.

We are aware of some server issues that are effecting Hi-mail this morning causing it to run very slowly or time out and return a 502 error. Our IT team are working on this now and hope to have it resolved shortly.

Hi-mail servers are now back up and running following a restart. Hi-mail will be running slower than normal until background processes have finished running but we expect this to improve shortly.

We are aware of server issues that are effecting Hi-mail. Our IT team are working on this now and hope to have it resolved shortly.

Hi-mail servers have been restored and all services are back up and running once again

Hi-mail servers are down due to a Database connection error. Our third party IT company have been notified and we’re waiting to hear back from them.

The server has been restarted and the connection restored. Hi-mail is now back up and running.

One of the FDM servers has lost connection which is causing issues with Hi-mail. The issue is being worked on now and we hope to have it resolved soon.

Service has now been fully resumed and Hi-mail is back up and running.

This morning we have found an issue with several servers not responding. Engineers are working on this problem now and we hope to have it resolved soon.

The copying of the drives and re-activation of Hi-mail has been successful. Hi-mail is now back up and running.

The Hi-mail Dashboard, Admin Portal, Virtual Printer and Polling Mode have all been tested and documents are arriving successfully on the server.

Please accept our sincere apologies for the disruption this has caused over the last few days. We are actively looking at ways of making sure this does not happen again in the future.

Update: 18:15pm
Following the new server installation, we found several errors when Hi-mail started up which are due to how the database interacts with the file storage system.

The Hi-mail database and file storage system need to be synced in order for the system to function however the backup of the database (DB) was taken before the system failed and therefore not in-sync with the file storage system.

We have managed to copy the required drives from the old server to a new drive which contain up to date (and synced) versions of both the DB and the file storage system and we have spent the day working to get these installed on the new server.

This has been a time-consuming process however as the file storage system is over 2TB in size and the data copying has to be performed over the FDM network without impacting daily opperations.

The drives have now been copied over to the new server and we will be re-activating Hi-mail shortly to test it.

Update: 6:50am
The new server has successfully been setup overnight and restored from the latest backup. Unfortunately we found some issues while testing Hi-mail overnight and need some more time to try and fix the issues.

There are 2 keys issues that need to be resolved:

1) Some accounts were unable to log in to Hi-mail due to a database error
2) Accounts that can login were unable to submit documents and received a timeout error when validating

We are working with our developers and engineers to try and restore full service as quickly as possible and will post another update when we have more information.

UPDATE: 14:40
The virtual server is taking longer than expected to create so the decision has been made to set-up a physical server to replace the broken hybrid server. This process is still likely to take several hours however and should be completed at some point tonight / early hours of tomorrow morning.

As such, Hi-mail will not be back online for the reminader of today (Wednesday 2nd March) however we expect it to be back up and running tomorrow morning.

UPDATE: The effected server has been worked on all evening however the issue is still not resolved. Engineers are now creating a new virtual space which they hope to be able to launch in the next couple of hours.

Unfortunately a Windows update has failed overnight on one of the Hi-mail servers. This update is being reverted and we hope to have the service back online as soon as possible.

Hi-mail is now back up following a server reboot. All services are now back up and running as usual.

Update: It appears that another Windows update has caused one of the new servers to stop responding and it requires a manual restart whch is being done now. We hope to have everything back up and running shortly.

The Hi-mail service appears to be unresponsive this morning. We are looking into the cause and will update this ticket again shortly.

Following a server restart, Hi-mail is now back up and running.

We have received reports of issues loading Hi-mail this morning which we’re investigating now. It appears that one of the servers is not responding and may require a re-start.

Hi-mail is now back up and running following a server reboot.

One of our Hi-mail servers has stopped responding. Our IT team are in touch with a third party company to resolve the issue.

All Hi-mail services have been restored following a server restart.

We have received reports of issues loading Hi-mail this morning which we’re investigating now. It appears that one of the servers is not responding our team is looking into this now.

Hi-mail services have now been restored and is back up and running.

A Hi-mail service has unexpectedly stopped responding affect some users connecting to Hi-mail. Our IT team have restarted this and Hi-mail should be up and running shortly.

Hi-mail services have now been restored.

Users have reported performance related issues with Hi-mail and we’ve restarted our servers to fix any underlying issues. Hi-mail should be back up and running again shortly.

Hi-mail is now back to normal and working as expected.

Hi-mail has become very slow this morning and we are investigating the cause for this now. You may experience some intermittent downtime over the next few minutes.

Hi-mail is now back up and running. All services have now resumed as normal.

Hi-mail service has unexpectedly stopped responding and our IT team are looking into this.

Hi-mail servers have been restarted and they are now up and running once again.

Some users have reported issues with Hi-mail loading this morning and we’re investigating this now.

Hi-mail is now back up and running as normal.

Part of the Hi-mail service has become unresponsive and requires a restart. Our IT team are working on this now and we will provide an update shortly.

Hi-mail servers are now back up and running following a server reboot.

Our Team have identified an issue with the Hi-mail database server. They’re working with our third party IT company to get Hi-mail back up and running as quickly as possible.

Hi-mail V1 is back up and running as normal.

Hi-mail services have gone down and our IT team are currently looking into this.

UPDATE 14:52: Our IT team have identified a problem with the network connection to one of our the servers earlier today. This has been escalated to our 3rd partly IT company who are currently looking into this. We hope to update you soon as we learn more on the outage.