Duration: 2 hours

Affected Components: Produce, Plan, Place, Point, Prove

Aug 21, 13:43:23 GMT+0
Identified - We will be performing a scheduled infrastructure upgrade to enhance system security, stability, and overall performance. **What to expect:** We expect minimal to no impact on your service. You may experience a brief interruption lasting a few minutes during this window as traffic is routed to updated nodes. Active sessions should automatically reconnect right after. Thank you for your patience and understanding!.

Sep 8, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 8, 19:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: Produce, Plan, Place, Point, Prove

Jul 28, 12:33:24 GMT+0
Identified - We will be performing a scheduled infrastructure upgrade to enhance system security, stability, and overall performance. **What to expect:** We expect minimal to no impact on your service. You may experience a brief interruption lasting a few minutes during this window as traffic is routed to updated nodes. Active sessions should automatically reconnect right after. Thank you for your patience and understanding!.

Aug 12, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 12, 10:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: Produce, Plan, Place, Point, Prove

Jul 28, 08:25:54 GMT+0
Identified - We will be performing a scheduled infrastructure upgrade to enhance system security, stability, and overall performance. **What to expect:** We expect minimal to no impact on your service. You may experience a brief interruption lasting a few minutes during this window as traffic is routed to updated nodes. Active sessions should automatically reconnect right after. Thank you for your patience and understanding!.

Aug 11, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 11, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: Produce, Plan, Place, Point, Prove

Jul 22, 07:20:07 GMT+0
Identified - We will be performing routine database maintenance to enhance system performance and security. **What to expect:** We expect minimal to no impact on your service. You may experience a **brief interruption lasting a few minutes** during this window as we switch over to the updated system. Active sessions should automatically reconnect right after. Thank you for your patience!.

Jul 29, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 29, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours and 8 minutes

Affected Components: Produce, Plan, Place, Point, Prove

Jul 1, 09:18:44 GMT+0
Investigating - We are aware of a critical issue causing widespread service unavailability across our platform. Our team is urgently investigating the situation to restore functionality. Next update will be provided within 30 minutes..

Jul 1, 09:34:35 GMT+0
Identified - We have identified and mitigated the underlying infrastructure issue that caused the service disruption. Most services are now recovering and coming back online. We are closely monitoring the system performance to ensure full stability. Thank you for your patience..

Jul 1, 09:45:13 GMT+0
Resolved - We would like to apologize for the temporary service unavailability experienced by our users earlier today. The disruption was caused by an unexpected performance bottleneck within our internal search and indexing infrastructure, which led to a cascade failure across our core application services. Once the root cause was identified, our engineering team successfully isolated the affected component, stabilized the data indexing layer, and initiated a controlled recovery process. All services have now been safely restored to full operational capacity. We are currently reviewing our infrastructure architecture and monitoring alerts for this specific component to prevent similar occurrences in the future. Thank you for your continued patience and understanding..

Jul 1, 10:46:56 GMT+0
Identified - We are currently experiencing degraded performance and increased response times across our platform. Our engineering team is actively expanding our core infrastructure capacity and scaling our internal services to alleviate the system load and **restore normal performance**. We sincerely apologize for the recurrence of this issue and will keep you updated here..

Jul 1, 11:27:14 GMT+0
Resolved - The performance issues have been mitigated, and all services have currently returned to standard operational parameters. Our engineering team has completed the infrastructure reconfiguration, and the expanded capacity (additional nodes, CPU, and memory) has successfully relieved the system bottleneck. While the environment is now stable, we are keeping the platform under intensive monitoring over the next few hours to ensure long-term stability and prevent any recurrence. We apologize for the disruption experienced today and thank you for your continued patience..

Duration: 1 day, 15 hours and 1 minute

Affected Components: Produce, Third Party: Amazon Web Services – AWS → AWS cloudfront, Place, Point, Plan, Prove, Web Services, Third Party: Amazon Web Services – AWS → AWS route53

Jun 22, 12:54:31 GMT+0
Investigating - We are currently experiencing high load on our core database systems, which may result in slow response times or intermittent unavailability across some of our services. Our engineering team is actively investigating the root cause and working on stabilizing the environment. We sincerely apologize for any inconvenience this may cause. We will provide updates on this page as more information or an Estimated Time of Resolution (ETR) becomes available..

Jun 22, 13:19:27 GMT+0
Identified - Access to the majority of our services should now be restored, and performance is returning to normal levels. While systems are currently stable, our engineering team is continuing to thoroughly investigate the root cause of the database load to ensure long-term stability and prevent this from happening again. We will provide a final update once the investigation is complete and the incident is officially resolved. Thank you for your patience..

Jun 22, 13:33:31 GMT+0
Resolved - The database load issues have been successfully mitigated, and all services are now fully operational. We are continuously monitoring the system's performance to ensure stability. A post-incident review will be conducted internally to prevent future occurrences. Thank you for your patience and understanding while we resolved this issue..

Jun 22, 14:33:01 GMT+0
Postmortem - We would like to apologize for the intermittent service degradation and connectivity issues experienced by some of our users earlier today. During a background system optimization process initiated late last week, an unexpected accumulation of data occurred within our core storage environment over the weekend. This led to temporary system congestion and performance slowdowns today. Our engineering team proactively managed system traffic and expanded infrastructure capacity to fully stabilize the environment. All services were safely restored to full operational capacity within 120 minutes. We are adjusting our system maintenance protocols to ensure that background tasks do not impact performance in the future. Thank you for your patience and understanding..

Duration: 3 hours

Affected Components: Produce, , Place, Point, Plan, Prove, , Papirfly Cloud Services → Cloud Services →

Jan 6, 07:00:00 GMT+0
Identified - Tactic app will undergo planned maintenance to perform a necessary system upgrade. The app may be temporarily unavailable during this period..

Jan 6, 07:00:00 GMT+0
Identified - Maintenance is now in progress..

Jan 6, 10:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 3 hours

Affected Components: Produce, , Place, Point, Plan, Prove, Papirfly Cloud Services →

Dec 11, 07:00:00 GMT+0
Identified - Tactic app will undergo planned maintenance to perform a necessary system upgrade. The app may be temporarily unavailable during this period..

Dec 11, 07:00:00 GMT+0
Identified - Maintenance is now in progress..

Dec 11, 10:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 7 hours and 42 minutes

Affected Components: Web Services

Nov 18, 11:48:00 GMT+0
Identified - Cloudflare is experiencing a global outage, which has caused many of our Brand Portal sites to become unstable..

Nov 18, 19:30:00 GMT+0
Resolved - Cloudflare's issues have been resolved.

Nov 18, 15:38:00 GMT+0
Monitoring - We have implemented a solution to move common services away from Cloudflare, and most of our Brand Portal sites are now operational. However, sites with customer URLs that are directly configured through Cloudflare remain dependent on Cloudflare's operational status..

Duration: 2 hours

Affected Components: Produce, , Place, Point, Plan, Prove, , Papirfly Cloud Services → Cloud Services →

Jun 29, 17:00:00 GMT+0
Identified - Maintenance is now in progress..

Jun 29, 17:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time to update the AWS K8s cluster to the latest version. During this maintenance period, there may be brief disruptions in connectivity as the cluster is updated. The availability of images and video files will not be disrupted. We apologize for any inconvenience this may cause, but this maintenance is necessary to ensure the ongoing security and reliability of the K8s cluster..

Jun 29, 19:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour

Affected Components: Produce, , Place, Point, Plan, Prove, Papirfly Cloud Services →

May 17, 17:00:00 GMT+0
Identified - Maintenance is now in progress..

May 17, 17:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time to update the AWS RDS cluster to the latest version During this maintenance period, there may be brief disruptions in connectivity as the cluster is updated. The availability of images and video files will not be disrupted. We apologize for any inconvenience this may cause, but this maintenance is necessary to ensure the ongoing security and reliability of the RDS cluster..

May 17, 18:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 4 hours and 40 minutes

Affected Components: Web Services

Jan 14, 09:50:00 GMT+0
Investigating - ImageVault has performance issues causing the site to operate very slowly..

Jan 14, 10:00:00 GMT+0
Identified - The team is working on fixing the issue and we will get back with an update as soon as possible..

Jan 14, 14:30:00 GMT+0
Resolved - All customers who have experienced the problem and have contacted us should have received a temporary solution..

Jan 16, 14:00:00 GMT+0
Postmortem - This is an update regarding the ImageVault performance. The reason to the downtime was that we have been migrated to Azure Front Door which unfortunately ImageVault was not fully compatible with. This has meant that the cache storage does not work in the same way. All customers who have experienced the problem and have contacted us should have received a temporary solution. Since yesterday (Wednesday, 15 January 2025) we also have a permanent correction developed, and there is a new version of ImageVault including this correction. We will upgrade each customer to provide the permanent solution. We are working as quickly as we can with the upgrade to ensure that everything is resolved. You will be updated continuously through your current case, but if you have questions or follow-up problems, we refer you first to our support, which you can easily reach by responding to the case or emailing [support@papirfly.com](mailto:support@papirfly.com)..

Duration: 1 hour

Affected Components: Produce, , Place, Point, Plan, Prove, Papirfly Cloud Services →

Aug 14, 21:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time on the AWS RDS cluster to reload CA certificates. During this maintenance period, there may be brief disruptions in connectivity as the certificates are updated. The availability of images and video files will not be disrupted. We apologize for any inconvenience this may cause, but this maintenance is necessary to ensure the ongoing security and reliability of the RDS cluster..

Aug 14, 21:00:00 GMT+0
Identified - Maintenance is now in progress..

Aug 14, 21:59:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 3 hours

Affected Components: Web Services

Jun 11, 08:00:00 GMT+0
Investigating - Search function unstable.

Jun 11, 09:00:00 GMT+0
Monitoring - We are now monitoring the situation to ensure we pick up any other issues that may arise..

Jun 11, 11:00:00 GMT+0
Resolved - This incident has been resolved..

Jun 11, 08:30:00 GMT+0
Identified - Some sites may experience reduced performance over the next few hours as the search indexes are being rebuilt..

Duration: 8 hours and 9 minutes

Affected Components: , Web Services, Legacy Brand Portals →

Dec 8, 17:22:00 GMT+0
Resolved - This incident has been resolved..

Dec 8, 09:15:00 GMT+0
Identified - We've identified an issue with one of our API endpoints, and it is currently not responding as expected. Our dedicated development team is actively investigating the issue to pinpoint the root cause and implement a swift resolution..

Dec 8, 09:45:00 GMT+0
Identified - We identified a critical issue within our AWS cluster, resulting in the failure of pods to start as expected. Our technical team is actively working on diagnosing and resolving the cluster-related issue to restore normal operation..

Dec 8, 09:13:00 GMT+0
Investigating - We regret to inform you that our AWS Cloud environment is currently experiencing an unexpected outage, impacting several services crucial to our operations. The affected features include Banner, Landing Page, and sending bulk/mass emails. > **_Only customers utilizing these functionalities may be affected._**.

Duration: 19 days and 15 hours

Affected Components: , , , , Cloud Services → Legacy Services → Legacy Brand Portals → Papirfly Cloud Services →

Nov 28, 15:00:00 GMT+0
Identified - # **Announcement** On **2023-12-18 at 06:00 CET**, Papirfly will end support of cipher suites using the Cipher Block Chaining (CBC) mode of operation on hosting environments. These cipher suites are known to be susceptible to attacks such as _padding oracle attack_, which can lead to data leaks and other security issues. For a list of TLS protocols and cipher suites that will be supported on each hosting environment after the above date, please consult the relevant section below. # **How does this affect you?** ## Browser support Most modern browsers, including Google Chrome, Microsoft Edge, Mozilla Firefox and Apple Safari support strong ciphers. We recommend updating your browser to the most recent version available. ## Applications and API access Check that support for strong ciphers, such as GCM, is enabled in your application. # Cloud hosting environment (AWS) All protocols and ciphers listed under the **TLSv1.2\_2021 policy** column in the [Supported protocols and ciphers between viewers and CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/secure-connections-supported-viewer-protocols-ciphers.html) document will be supported on our cloud hosting environment (AWS). # On-prem hosting environment (Lan-x) The protocols and corresponding cipher suites listed below will be supported in our on-prem production environment. ## Supported TLS protocols * TLSv1.2 ## Supported ECDSA ciphers * TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384 * TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256 ## Supported RSA ciphers * TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384 * TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256 # H ave any questions or need help? C ontact: [Papirfly Support](mailto:support@papirfly.com?Subject=Ending-support-for-CBC-ciphers-in-TLS-connections-to-Papirfly-services).

Dec 18, 06:00:00 GMT+0
Resolved - This is now in effect..

Duration: 8 hours and 30 minutes

Affected Components: , Legacy Brand Portals →

Sep 8, 09:00:00 GMT+0
Investigating - Some customers are reporting degraded performance and issues with some Papirfly modules..

Sep 8, 09:30:00 GMT+0
Identified - Unnusal high load on one of our database servers caused by a running import job. We are working on minimizing the impact of the degraded performance. Some Papirfly websites may experience degraded performance and issues with some Papirfly modules while this job is running..

Sep 8, 11:30:00 GMT+0
Monitoring - Still high load on the affected database server..

Sep 8, 11:00:00 GMT+0
Monitoring - Still high load on the affected database server..

Sep 8, 17:30:00 GMT+0
Resolved - Performance is back to normal..

Duration: 2 hours and 6 minutes

Affected Components: , Legacy Brand Portals →

Aug 24, 07:45:00 GMT+0
Identified - Performance test started.

Aug 24, 07:00:00 GMT+0
Identified - A performance test has been scheduled for a Papirfly service that may for a limited period affect performance for some customers. D uring the test we will constantly monitor our systems and a bort the test if major performance degradation is detected..

Aug 24, 09:51:00 GMT+0
Completed - Performance test is complete.

Duration: 3 hours and 57 minutes

Affected Components: , Legacy Brand Portals →

Nov 14, 06:30:00 GMT+0
Investigating - We are experiencing problems with our database servers in one of our production environments making some of our Papirfly websites inaccessible..

Nov 14, 08:11:00 GMT+0
Identified - Problems with the uploading of assets were found and a fix has been applied, these issues were caused by problems with our image and video processing services..

Nov 14, 09:27:00 GMT+0
Resolved - All systems operational.

Nov 14, 07:04:00 GMT+0
Identified - We have identified the issue and a fix has been applied..

Nov 14, 09:18:00 GMT+0
Monitoring - There may still be some degraded performance due to these recent problems, but we are constantly monitoring to ensure we pick up any other issues that may arise..

Nov 14, 07:55:00 GMT+0
Investigating - We have detected additional issues causing degraded performance and problems with uploading assets..

Nov 14, 14:00:00 GMT+0
Postmortem - ## Summary of the problems affecting access to some Papirfly Web Sites The issues were caused by inaccessible databases and processing services in one of our production environments after a scheduled update. * 06:15: Our systems were updated automatically (during a scheduled maintenance window) which caused some databases and processing services to be inaccessible * 07:30: Problems with access to some Papirfly Web sites were detected by the first System Administrator arriving at the office * 08:00: The issue causing inaccessible databases were identified and a fix applied * 09:05: The issue causing problems with processing services were identified and a fix applied * 09:10: Systems operational with degraded performance due to the processing backlog * 10:30: Performance issues resolved and all systems operational > Note that time is in CET.

Nov 1, 14:00:00 GMT+0
Resolved - # Introduction Details of a critical vulnerability in OpenSSL will be published on 1st November 2022 between 1300 and 1700 UTC \[[1](https://mta.openssl.org/pipermail/openssl-announce/2022-October/000238.html)\]. The vulnerability only affects version 3.x of OpenSSL. OpenSSL is a widely used library for encrypting network traffic. OpenSSL rates the vulnerability as critical \[[2](https://www.openssl.org/policies/general/security-policy.html)\]. Papirfly has no more information about the vulnerability than what OpenSSL itself has published. # What we have done * Reviewed all Pairfly services available on the public Internet to find any potential usage of OpenSSL 3.x * Prepared to patch any services that may be vulnerable * Prepared to take any services offline until they are patched, if they are determined to be vulnerable # What we have found * None of our services use OpenSSL 3.x # What we are doing * Waiting for OpenSSL to publish details of the vulnerability # References * \[1\] <https://mta.openssl.org/pipermail/openssl-announce/2022-October/000238.html> * \[2\] <https://www.openssl.org/policies/general/security-policy.html>.

Duration: 40 minutes

Affected Components: , Legacy Brand Portals →

Oct 6, 13:49:00 GMT+0
Investigating - Some Papirfly services started responding slower than usual. It appears to be caused by network instability. We are investigating together with our hosting provider..

Oct 6, 14:20:00 GMT+0
Monitoring - All Papirfly services are working normally again. We are following the situation and investigating the root cause of the earlier disruption..

Oct 6, 14:29:00 GMT+0
Resolved - All Papirfly services are working normally..

Oct 7, 11:00:00 GMT+0
Postmortem - Our hosting provider checked their network for anomalies and found signs of resource limits in a firewall. The firewall logs have been sent to the vendor for further analysis and the firewalls were upgraded and restarted during a planned maintenance window on Thursday evening. The network instability occurred between 15:48:40 and 16:20:40 CEST on Thursday 2022-10-06..

Affected Components: , , Legacy Services → Legacy Brand Portals →

Dec 15, 09:30:00 GMT+0
Resolved - # Introduction On 10th December 2021, a critical vulnerability CVE-2021-44228 - referred to as _Log4Shell_ \- was disclosed. Many large software companies and online services are affected such as Amazon, Apple iCloud, Cisco, ElasticSearch, Tesla, Twitter and many more. This article summarizes the results of our investigation to date. # Description The vulnerability is caused by Apache Log4j which is a Java-based logging library used in many products and applications. The vulnerability allows an attacker to execute their own code on a remote server, a so-called Remote Code Execution (RCE) and potentially take full control of the system. Services and systems that use Apache log4j library between versions 2.0 and 2.14.1 are affected. # What have Papirfly done? We scanned our infrastructure to identify vulnerable applications and services. Here is a list of our findings and actions taken: * None of Papirfly's Internet-facing applications use Log4j directly. * We found one application that includes an older version of Log4j (pre 2.0) that is not affected by this vulnerability. * We found one application, Elasticsearch 7.12.0, that includes Log4j 2.11.1 that is affected by this vulnerability. This application is not available from the public Internet and is only sent sanitised input from the Papirfly service, so we do not believe an attacker can exploit the vulnerability against Elasticsearch. Nevertheless, out of an abundance of caution, we have taken the following actions: * 2021-12-13 09:10:00: Following advice from the authorities and the vendor, we implemented configuration changes and restarted services to immediately mitigate the vulnerability. * 2021-12-13 11:30:00: Following advice from the vendor, we upgraded to Elasticsearch 7.16.1 which disables JNDI lookups and includes a patched version of Log4j where the JndiLookup class has been removed. * 2021-12-15 08:30:00: Our hosting provider informed us that we are using a VMware component called _vCenter_ (not Internet-facing) in our on-prem production environment that uses a vulnerable version of Log4j. They are updating our vCenter server appliance from 6.7 to 7.0 U3A. * 2021-12-15 12:01:00: A further vulnerability (CVE-2021-45046) was disclosed on 14th December after it was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations. This does not affect the Papirfly infrastructure as we have upgraded to Elasticsearch 7.16.1 which mitigates the Log4j vulnerability by removing the JndiLookup class from the classpath. * 2021-12-20 10:00:00: A further vulnerability (CVE-2021-45105) was disclosed on 17th December after it was found that Log4j 2.16 was vulnerable to another DoS vulnerability. This does not affect the Papirfly infrastructure as we have upgraded to Elasticsearch 7.16.1 which provide full protection against all known CVEs. To prevent false positive alerts in vulnerability scanners that look at only the version of the Log4j dependency, and in the interest of compliance, we are in the process of upgrading to Elasticsearch 7.16.2\. This upgrades Apache Log4j2 to version 2.17.0 and retains the mitigations delivered in 7.16.1. We continue to monitor the situation and follow recommendations from the authorities and our vendors. # Conclusion We do not believe that attackers have managed to exploit the vulnerability. We have taken steps as recommended by the authorities and our vendors to further secure Papirfly services. We are continuing to investigate our exposure to this vulnerability and will provide further updates if any new risk to our users or our products is identified. We are ready to handle any incidents. # Further information * [Norwegian National Security Authority (NSM)](https://nsm.no/fagomrader/digital-sikkerhet/nasjonalt-cybersikkerhetssenter/varsler-fra-ncsc/utvidet-oppdatering-for-apache-log4j-cve-2021-44228) * [Apache Foundation Log4j mitigation methods](https://logging.apache.org/log4j/2.x/security.html) * [Elasticsearch B.V. Security Announcement](https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476).