Resolved -
Over the last several days, the team has monitored diligently and in partnership with our fax carriers made several adjustments to resolve intermittent fax delays. Today, in partnership with our fax carriers, we have seen normal fax delivery times through morning and peak hours and consider the incident resolved. Please see the earlier update for the RCA and should you have questions or additional issues to report, please contact Partnersupport@updox.com.
Jun 4, 16:00 EDT
Update -
Incident Overview
From May 27 through May 30, 2025, Updox experienced service disruptions due to recurring firewall failures
at one of our data centers. Updox has redundant firewalls to ensure continued operations but both firewalls
experienced failures intermittently. These failures impacted platform availability and caused cascading
effects across multiple services.
How We Responded
Our engineering team worked diligently to identify and resolve the root cause of the disruptions. On May 27,
the earliest signs of the issue emerged when faxing services were disrupted due to network storage problems
triggered by firewall instability.
As the disruptions continued, we identified recurring failures in the firewall hardware at one of our data
centers as the root cause. These failures intermittently interrupted network traffic, impacting both platform
availability and dependent services.
Throughout the incident, our team collaborated closely with the data center to restart the firewalls during
each occurrence. We also made additional adjustments to stabilize the platform and closely monitored
system performance to minimize user impact. To fully resolve the issue, we replaced the failing firewall
hardware with new equipment during a planned maintenance window on May 30 between 10:00 PM and
12:00 AM.
Corrective and Preventative Actions
To ensure these issues do not happen again, we are taking the following steps:
• Firewall Replacement: The faulty firewall hardware was replaced during a planned maintenance
window on May 30, 2025, between 10:00 PM and 12:00 AM. This upgrade restored stable network
performance and resolved the root cause of the disruptions.
• Post-Cutover Monitoring: We are actively monitoring network and system performance metrics
following the firewall replacement to ensure continued stability and responsiveness. During this
monitoring phase, we are partnering with our fax carriers to fine tune our network peering together to
resolve temporary fax delays.
We know that many of you rely on Updox every day to support your operations. We sincerely apologize for this
disruption and are committed to strengthening our systems to prevent it from happening again. Thank you for
your patience and continued trust.
Jun 3, 17:47 EDT
Update -
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
Jun 3, 11:39 EDT
Update -
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
Jun 2, 15:45 EDT
Update -
We have completed the planned maintenance and experienced a 3 minute temporary outage during the cutover. Thank you for your patience. The team continues to monitor performance and networking prior to marking the incident resolved.
May 31, 00:06 EDT
Update -
Thank you for your patience. To permanently resolve the intermittent outages caused by the faulty hardware, we will be performing a hardware upgrade during a scheduled maintenance window.
Maintenance Window: May 30th 10.00PM ET to 30th May 12.00AM ET
Expected Impact: Temporary service unavailability
Our team is working to complete this upgrade as quickly and safely as possible to restore full reliability. We appreciate your continued understanding and will provide updates once the maintenance is complete.
May 30, 14:02 EDT
Update -
We’re aware of the temporary outage that has been occurring every few hours since yesterday. This downtime is due to a faulty hardware component, and our team is addressing it with urgency. We understand how important uptime is to you and apologize for the inconvenience.
May 30, 10:02 EDT
Update -
We’re aware of the temporary outage that has been occurring every few hours since yesterday. This downtime is due to a faulty hardware component, and our team is addressing it with urgency. We understand how important uptime is to you and apologize for the inconvenience.
May 29, 12:46 EDT
Update -
We have implemented a fix for the disruption in service and the issue is resolved in production. All systems are operational at this time in production. We will continue to monitor. We recognize the same issue has occurred several times this week and apologize for the inconvenience. We will share more information and timelines shortly but are working to replace network hardware causing issues as fast as possible.
May 29, 11:30 EDT
Monitoring -
We have implemented a fix for the disruption in service and the issue is resolved in production. All systems are operational at this time. We will continue to monitor. We recognize the same issue has occurred several times this week and apologize for the inconvenience. We will share more information and timelines shortly but are working to replace network hardware causing issues as fast as possible.
May 29, 09:50 EDT
Identified -
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
May 29, 09:00 EDT
Monitoring -
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
May 28, 12:54 EDT
Update -
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
May 28, 12:38 EDT
Identified -
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
May 28, 12:35 EDT
Investigating -
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
May 28, 11:59 EDT
Monitoring -
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
May 28, 09:31 EDT
Identified -
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
May 28, 09:13 EDT
Update -
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
May 28, 09:06 EDT
Investigating -
Updox is experiencing a disruption in service. Users may not be able to access Application API's are unavailable as well. . We are investigating the issue and will post updates every 30 minutes.
May 28, 08:56 EDT
