Over the last several days, the team has monitored diligently and in partnership with our fax carriers made several adjustments to resolve intermittent fax delays. Today, in partnership with our fax carriers, we have seen normal fax delivery times through morning and peak hours and consider the incident resolved. Please see the earlier update for the RCA and should you have questions or additional issues to report, please contact Partnersupport@updox.com.
Posted Jun 04, 2025 - 16:00 EDT
Update
Incident Overview
From May 27 through May 30, 2025, Updox experienced service disruptions due to recurring firewall failures at one of our data centers. Updox has redundant firewalls to ensure continued operations but both firewalls experienced failures intermittently. These failures impacted platform availability and caused cascading effects across multiple services.
How We Responded
Our engineering team worked diligently to identify and resolve the root cause of the disruptions. On May 27, the earliest signs of the issue emerged when faxing services were disrupted due to network storage problems triggered by firewall instability.
As the disruptions continued, we identified recurring failures in the firewall hardware at one of our data centers as the root cause. These failures intermittently interrupted network traffic, impacting both platform availability and dependent services.
Throughout the incident, our team collaborated closely with the data center to restart the firewalls during each occurrence. We also made additional adjustments to stabilize the platform and closely monitored system performance to minimize user impact. To fully resolve the issue, we replaced the failing firewall hardware with new equipment during a planned maintenance window on May 30 between 10:00 PM and 12:00 AM.
Corrective and Preventative Actions
To ensure these issues do not happen again, we are taking the following steps:
• Firewall Replacement: The faulty firewall hardware was replaced during a planned maintenance window on May 30, 2025, between 10:00 PM and 12:00 AM. This upgrade restored stable network performance and resolved the root cause of the disruptions.
• Post-Cutover Monitoring: We are actively monitoring network and system performance metrics following the firewall replacement to ensure continued stability and responsiveness. During this monitoring phase, we are partnering with our fax carriers to fine tune our network peering together to resolve temporary fax delays.
We know that many of you rely on Updox every day to support your operations. We sincerely apologize for this disruption and are committed to strengthening our systems to prevent it from happening again. Thank you for your patience and continued trust.
Posted Jun 03, 2025 - 17:47 EDT
Update
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
Posted Jun 03, 2025 - 11:39 EDT
Update
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
Posted Jun 02, 2025 - 15:45 EDT
Update
We have completed the planned maintenance and experienced a 3 minute temporary outage during the cutover. Thank you for your patience. The team continues to monitor performance and networking prior to marking the incident resolved.
Posted May 31, 2025 - 00:06 EDT
Update
Thank you for your patience. To permanently resolve the intermittent outages caused by the faulty hardware, we will be performing a hardware upgrade during a scheduled maintenance window.
Maintenance Window: May 30th 10.00PM ET to 30th May 12.00AM ET Expected Impact: Temporary service unavailability
Our team is working to complete this upgrade as quickly and safely as possible to restore full reliability. We appreciate your continued understanding and will provide updates once the maintenance is complete.
Posted May 30, 2025 - 14:02 EDT
Update
We’re aware of the temporary outage that has been occurring every few hours since yesterday. This downtime is due to a faulty hardware component, and our team is addressing it with urgency. We understand how important uptime is to you and apologize for the inconvenience.
Posted May 30, 2025 - 10:02 EDT
Update
We’re aware of the temporary outage that has been occurring every few hours since yesterday. This downtime is due to a faulty hardware component, and our team is addressing it with urgency. We understand how important uptime is to you and apologize for the inconvenience.
Posted May 29, 2025 - 12:46 EDT
Update
We have implemented a fix for the disruption in service and the issue is resolved in production. All systems are operational at this time in production. We will continue to monitor. We recognize the same issue has occurred several times this week and apologize for the inconvenience. We will share more information and timelines shortly but are working to replace network hardware causing issues as fast as possible.
Posted May 29, 2025 - 11:30 EDT
Monitoring
We have implemented a fix for the disruption in service and the issue is resolved in production. All systems are operational at this time. We will continue to monitor. We recognize the same issue has occurred several times this week and apologize for the inconvenience. We will share more information and timelines shortly but are working to replace network hardware causing issues as fast as possible.
Posted May 29, 2025 - 09:50 EDT
Identified
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
Posted May 29, 2025 - 09:00 EDT
Monitoring
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
Posted May 28, 2025 - 12:54 EDT
Update
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
Posted May 28, 2025 - 12:38 EDT
Identified
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
Posted May 28, 2025 - 12:35 EDT
Investigating
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
Posted May 28, 2025 - 11:59 EDT
Monitoring
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
Posted May 28, 2025 - 09:31 EDT
Identified
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
Posted May 28, 2025 - 09:13 EDT
Update
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
Posted May 28, 2025 - 09:06 EDT
Investigating
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
Posted May 28, 2025 - 08:56 EDT
This incident affected: Interface / Connection to Updox (RESTful API, Web User Interface, Patient Portal, Updox QA development environment), Telehealth (Video Chat, Secure SMS texting, SMS texting, Video Chat SMS Invites), and Faxing, Reminders, Scheduling, Direct e-Mail Messaging, Updox Secure Messaging, Import to EHR, Webhooks.