How We Reset 4 Million Passwords

by Natasha Postolovski. Illustration by eightbit on Envato Studio.

In May of this year, Envato required every one of its ~4 million users to do a password reset. Each user could no longer log into their accounts on Envato Market or Studio until they completed the reset process. Both sites are thriving products, together serving up more than $200 million in financial transactions.

In this post, we’ll share the story of how we helped keep our user base secure without a significant impact on revenue, and the challenges and successes we faced throughout the process.

The Lead Up

On the 13th of October last year, Adobe reported that the account details of 2.9 million customers (later revealed to be up to 150 million customers) had been compromised. This was scary news for many, and for us at Envato. We wondered how many of our users, many of whom are creatives, would be sharing account details across both Adobe and Envato Market.

“We’re a creative marketplace and Adobe’s a creative product,” says Steve Hoy, Operations Director at Envato. “Over time, we cross-referenced how many people on that list also had an Envato account. Adobe is just about the worst company to be hacked for us.”

In April of this year, Heartbleed struck. Like many other businesses, we were following best practices by using SSL encryption on our accounts site. Yet, this meant we were vulnerable to Heartbleed. We upgraded OpenSSL, revoked and re-issued our SSL keys and certificates, and reset all user sessions. We also encouraged all users to change their passwords.

In the weeks following the request, we noticed that many users were leaving their pre-Heartbleed passwords unchanged and exposing themselves to risk of compromise. Meanwhile, we were also seeing an increase in suspicious access attempts on Envato Market, where hackers were gaining access to accounts on the first or second try. The activity suggested the use of a password list. With security concerns increasing, it was time to decide how Envato should respond.

The Reset Project

In May, the Envato Market team, led by CEO Collis Ta’eed, decided they had no choice but to do a reset of everyone’s passwords. The reset would mean users would lose access to their accounts until they created a new password that met Envato’s current password requirements.

The reset would involve three large projects requiring considerable effort and mobilisation. First, the UX team, in tandem with the development team, would need to develop a password reset flow that was simple, logical, and handled edge-cases. Second, users would need to be educated about the reset and the reasons behind it through various comms channels and an email alert. Third, the customer support team would need to brace itself for an increased enquiry load.

When these projects kicked-off, nobody fully understood how long they would last, and how complex they might be. “It was about 2 months of extra work,” says Steve Hoy.

Part 1: User Experience

Justin French heads up the UX team at Envato. It fell to him to craft a UX response to the security requirements, one that would be effective not just in this instance, and could be used again in future.

“We realised that this was something that needed to survive for quite a while, and the part that was about Heartbleed was really small. We almost took it out of the equation and it became something really generic,” he says. “We identify a reason for you to change your password, and when you have a reset flag on your account you get ushered into this temporary flow and then pop out on the other side, and then it’s happy days.”

One of the team’s biggest initial challenges was crafting a process that would give the user the information they needed to allay their fears. “If you tell me I need to reset my password, I’m going to have some instant anxieties about what that might mean. Has my account been compromised? What’s going on with my money? You need to go through that list of fears and think, what are the likely things the user will want to know? It’s not always as neat as the little flow diagram makes it out to be.”

Dealing with edge-cases is one of the most difficult parts of UX, and a project like this could easily be rife with them. For Justin French, a collaborative approach and lots of internal testing meant that most edge cases were identified and accommodated for in advance. He says that, had the process been different, some users may have slipped through the cracks.

“How does it fall apart when it doesn’t go that ideal, pristine, nicely mapped out three minute journey? Dealing with that is really just talking it through with people over and over again, which makes it better and better. There are no answers at my desk,” he says.

The project showed strengths in the way Envato’s UX team works with the development team to implement new user flows. First, they focused on creating something that worked as soon as possible, tweaking the details once the overall structure had been confirmed. The developers were able to work mainly with pre-existing components in the Envato Market Styleguide. “If we decided that we wanted to ship sooner, we would have had an end-to-end solution that we knew could be better, rather than having half of a perfect solution. That’s the most important thing on any project, getting to the point where you could ship it as fast as possible.”

“It went straight from wires to code, and any design work was done mostly in the browser. We used the existing building blocks as much as possible,” says Justin, a process he credits with speeding up the implementation of new user flows.

“We’re really far down that road where you can have a wireframe or even a thing in your head, and you can go find the right five classes in the Styleguide and get really close to something production-ready very, very fast.”

Justin says it is when you put yourself in the user’s shoes and cover all the possibilities that good UX work emerges. “When you deviate from the golden path is when you go from something that is kind of an OK experience to something that has been really thought through. Going through every step, looking for the anxieties, giving users the information they need; that’s what I enjoy most.”

  • Empathise with the user. What are they worried about right now? What information do they need to confidently move forward?
  • No answers at your desk. Talk to others and get lots of opinions on your work. This will make the flow better and ensure most edge cases are caught in advance.
  • Create a 'Styleguide' for your app. Developers can quickly get features up and running, and looking good, using pre-existing components and classes.
  • Have teams test their own work. Turn your staff and colleagues into an army of testers for critical pieces of work.

Part 2: Customer Support

A day or two after users were pushed through the password reset process, the team took stock of key metrics. “We’d worked out that for every 100 people who attempted to do the reset, 2 support tickets came in," says Steve Hoy. "That figure of 2% was static right throughout the project. It allowed us to track how many extra support tickets we could expect and how many extra people would need to come in to help.” With a thriving community of ~4 million users, the support team braced itself for an increased support load.

Kelly Dent is the Customer Experience Manager at Envato. She says that, surprisingly, it wasn’t a problem with the password reset flow that led to an increase in tickets, but a simple case of users having old, inactive email addresses associated with their Envato accounts. They couldn’t get into these old email accounts, and were therefore unable to complete the reset process. “That’s where the impact on support exploded,” she says. The support team were dealing with hundreds of additional tickets per day.

It became apparent that the team would need extra help to deal with the increased load. “I think we probably had temps around for about 8 weeks," she says. "With minimal training they were able to be effective immediately, because they focused only on password reset related tickets. It allowed the rest of the team to continue on with normal tickets.”

Restoring access to these user’s accounts required that they first had to prove their identity. With hundreds or thousands of dollars of credit available on user accounts, the support team couldn’t risk giving account access to someone making false claims about who they really were.

“We’d ask them to provide the account's username, current email, a full name, and usually a question about their last purchase. Plus, if we could, we’d verify their IP address against our records.” The process, though time consuming, helped ensure only account owners were gaining access to Envato accounts.

All told, the support team received an additional 8,200 support tickets related to the password reset, to which they sent 12,851 replies.

  • Brace yourselves. Major challenges for your product are likely to create an additional support load in ways you can't predict.
  • Be prepared to hire temporary staff to deal with the extra load. When temporary support staff join, have them double down on one type of ticket, which drastically minimizes the amount of training they'll need to receive.
  • When resetting passwords, prepare for old, inactive emails. If your reset flow follows the standard email token gateway step, prepare to hear from customers who can't access old or inactive email accounts. This is an even bigger problem on apps that have been around for longer than 5 years.
  • Create a dedicated email address for the issue. We created accountsecurity@envato.com and linked it to our support system. "It showed that we were looking at this seriously, that it was a specifically targeted message and a very special process," says Kelly.

Part 3: Email Delivery at Scale

It wasn’t enough to silently log all users out of Envato’s sites, and force a password reset when they attempted to log back in. The Envato Market team knew they needed to explain the reasons behind the actions they'd taken, and encourage users to quickly update passwords that were now potentially insecure. It was clear that they had to send a special once-off security email.

John Viner, Software Development Manager at Envato, says that, to his surprise, we'd never sent out a mass email to our all users. “There was nothing that we had set up to be able to do that,” he says. “We send email to our users all the time, but they’re trigged by their activity. We had never before needed to individually email users with a tailored or a generic message. This project was about working out the best way to do that.”

Envato typically uses MailChimp for its marketing emails, but these emails were different. “In MailChimp, you can only email people marketing material if they’ve opted in,” says John. “In any material that you send subscribers, you have to provide them with the opportunity to opt-out. This was a security-related email, not a marketing email. From a legal perspective, this meant that if a user opted-out we couldn’t then send them further security updates. That seemed wrong. We had to look at sending them emails via our transactional email tool, Mandrill."

Mandrill is a service for sending emails related to account activity and transactions. It seemed like the right tool for the job, but a couple of open questions remained. Would it cost tens of thousands of dollars to send so many emails? And could we send four million emails in one hit?

John Viner says one of the team’s biggest concerns was sending out such a large volume of emails without triggering spam filters. “The big issue was sending these emails and not impacting our transactional emails, and not impacting the reputation of our mass emails, so that we could continue to get through to as many people in the group of 4 million. We had to learn really quickly how Mandrill’s reputation system worked.”

The second complexity was working out who of Envato’s 4 million users would be emailed, and in what order. “It took some time to work out a top to bottom order, and to slice that at chunks, and for every chunk that we sent out, to track who completed the process. We wanted to email people based on those who’d most recently signed up, because we figured that people who signed up three years ago were more likely to not exist, or mark us as spam, or have forgotten about us.

"We sent small batches out, first of 1,000, then 5,000 then 50,000, and closely monitored our stats in Mandrill,” he says.

All in all, the team sent security-related emails to around 500,000 of Envato’s most active users while managing to maintain high rates of deliverability.

  • Send test emails to multiple people before sending to everyone. John says that a few mistakes and typos slipped into the first small batch of emails sent out. After testing them more thoroughly with a wider range of people, future emails were free of errors.
  • Guard your reputation. By being aware of how an email address's reputation affects its deliverability, we were able to email almost 500,000 people with few issues.
  • Use a dedicated tool. The success of the effort was a result of choosing the right tool for the job. If we'd used an email service designed for marketing campaigns, we would have struggled. A transactional email service like Mandrill was the perfect fit.

Security In Focus

“In addition to the mass password reset, we immediately established a dedicated security stream,” says John Viner. The stream brings together some of Envato’s smartest security minds to protect user accounts and information.

Steve Hoy says Envato’s ‘Helpful Hacker’ program has also made a difference. Well-intentioned hackers and security researchers who discover and report vulnerabilities in Envato Market are added to a public honor roll. “It’s about giving them some kudos in the hacker community and recognising hackers who help us out,” says Steve.

Envato were inspired by Google’s Vulnerability Reward Program and Github’s Bug Bounty, two successful programs designed to harness the skill of hackers for the greater good.

A New Best Practice?

As the number of popular web applications existing for 5+ years increases, security breaches will continue. In the past, companies have periodically asked users to refresh their passwords, with varying compliance rates.

Though a forced reset was a logistical challenge, knowing that all our active users are using updated passwords is reassuring for us, and for them. Our strategy team also found that there was no noticeable impact on revenue during the password reset project.

As hackers become more sophisticated and find new ways to distribute lists of passwords and details, it may become an industry best practice to force a refresh of user passwords every few years, particularly when an application deals with sensitive financial details.

We hope the challenges and successes of our experience will help guide other companies who are also experiencing mounting concerns about account security.

comments powered by Disqus