Pin Payments outage, August 2018: Intermittent issues with Dashboard and Charges
Pin Payments reported and resolved an incident: Intermittent issues with Dashboard and Charges. It lasted 1 minute.
OKResolved
From Pin Payments's official status page, in their words.
What happened
- Started
- Resolved
- Duration
- 1 minute
- Impact
- none
- Investigating
- Identifiednot posted
- Monitoringnot posted
- Resolved
Why it happened
Pin Payments has not given a cause in its updates. We never guess one.
How it was resolved
On the morning of 9 August 2018 between the hours of 03:30 - 08:30 AEST merchants using the Pin Payments platform may have experienced errors creating charges and accessing the dashboard. The problem occured during an Amazon Web Services (AWS) scheduled maintenance period targeting a node in a high availability cluster that constitutes part of the platform. We were aware of this maintenance and assessed at the time that the system would failover and utilise the remaining nodes in our high availability cluster as designed. The failover happened correctly but due to a misconfiguration elsewhere on platform the results were unexpected. As a Pin Payments merchant, you may have been affected by charges marked as failed in your Pin Payments account actually succeeding upstream at our acquiring bank. We have identified each occurence of this and refunded the funds back to the affected cardholders. There is nothing for you to do, except assure any customers who ask that the funds have been refunded. In mitigation we have modified our operating procedure, ensured the correct configuration is in place and added additional metrics and monitoring.
Timeline
Oldest first, from the first notice to the resolution. From Pin Payments's official status page, in their words.
- resolved
On the morning of 9 August 2018 between the hours of 03:30 - 08:30 AEST merchants using the Pin Payments platform may have experienced errors creating charges and accessing the dashboard. The problem occured during an Amazon Web Services (AWS) scheduled maintenance period targeting a node in a high availability cluster that constitutes part of the platform. We were aware of this maintenance and assessed at the time that the system would failover and utilise the remaining nodes in our high availability cluster as designed. The failover happened correctly but due to a misconfiguration elsewhere on platform the results were unexpected. As a Pin Payments merchant, you may have been affected by charges marked as failed in your Pin Payments account actually succeeding upstream at our acquiring bank. We have identified each occurence of this and refunded the funds back to the affected cardholders. There is nothing for you to do, except assure any customers who ask that the funds have been refunded. In mitigation we have modified our operating procedure, ensured the correct configuration is in place and added additional metrics and monitoring.
What this means for a business (and what to check now)
- Retry anything that failed during the 1 minute.
- Enter any payments, bookings or jobs you took another way.
- If a deadline fell inside the window, keep this page as your record of the outage.
The full guide: what to do when a service you rely on is down
Other Pin Payments incidents
- We are currently experiencing an upstream outage which is causing requests to fail. We are investigating the issue., resolved after 58 minutes
- Issue with processing charges, resolved after 1 minute
- Partial outage of charge creation, resolved after 2.1 hours
- Dashboard and Payment Page Outage Due to Cloudflare Issues, resolved after 2.6 hours
- Charge Processing Failures with American Express, resolved after 2.7 hours
Hear about the next Pin Payments incident as it starts
Free email alerts when the official status changes, with what to do while it is down.