Sponsored by: Sucuri: Incident Response, Monitoring, DDoS mitigation and WAF for websites

This has been a mega week with a couple of pretty contentious blog posts which frankly, are the best kind! It gets so boring when everyone just nods and agrees...

But seriously, the one on ad blockers in particular shows just what a mess we've gotten ourselves into and the "ban all the ads (or anything that has even a sniff of an ad)" proponents are a big part of the problem. I talk about it in detail in the video though so here it is, along with all the podcasts too:

iTunes podcast | Google Play Music podcast | RSS podcast

(And yes, that's a mic bottom left of frame, I recorded with my good boom mic this time and totally didn't see it in frame until I went to edit. Works fine on the podcast though!)

References

1. How responsible are companies when partners lose their data? (no, a checklist or even an audit won't stop a partner from publishing your DB backups to a publicly facing web server)
2. Ad blockers are part of the problem (holy shit some people lost their minds over this one...)
3. People are kinda sensitive about their personal data (yes, even if they've published it publicly, they have expectations about how it's used)
4. I've loaded some spam lists into HIBP (the 15% of people who didn't want this can be quite vocal...)
5. New Pluralsight course - "The Infosec Big Picture" (I love this course, just read the blog post or even better, watch the course!)

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

It's hard to believe it, but Sunday 4 December will mark 3 years since I launched Have I been pwned. A huge amount has happened in that time, not just for HIBP but for the industry and indeed for me personally. I certainly didn't expect it to become what it is, not in terms of the amount of data or the number of people visiting and subscribing and certainly not the media attention it's drawn from all over the world. That's posed some really unique challenges but been enormously rewarding too.

To celebrate, I thought I'd do a live streamed "Ask Me Anything" next week. I want to stream it so that I can answer questions verbally and show things via screen share. If people have questions that I can illustrate by demonstration, then that's a much better experience for all than me writing text-based answers in a Reddit-style AMA. I'm going to run it via YouTube Live's streaming service and it'll happen at 06:00 Tuesday 6 December my time on the Gold Coast Australia. That's an hour later in our other eastern Australian states, 20:00 Monday 5 in UTC time, 15:00 on the east US coast and 12:00 midday on the west. Everything else, well, you can work out for your own time zones but to make things super easy, here's an iCalendar file you can add to your calendar. You can also watch the live stream here and it will be viewable in full immediately once run as well:

You can ask me anything in two different ways:

1. Use the comments section below in advance of the live stream. I'd really like to get a bunch of them beforehand and I'll come prepared with answers and demos and anything else useful.
2. Go to the video on YouTube and use the comments section to the right of the video (you'll need a Google account to comment). Ask me anything there during the live stream.

To help everything run smoothly, I've enlisted my good mate Scott Helme to help moderate so you might see him pop up during the event (you may remember Scott from such projects as report-uri.io, securityheaders.io and that time he got his car hacked from the other side of the world).

There's a bunch of questions I expect I'll get around the following:

1. How I get data
2. Things learned from those trading it
3. If I've had any legal issues
4. The cost of running it
5. Scale, performance and other cloud things
6. Why I keep posting sunny photos of Australia

But seriously, nothing is off limits, ask me anything and I'll answer whatever I can.

Last thing - at the end of this event I'm going to be giving everyone a very large whack of data that will give you much better insight into what's going on with these breaches. I'll save it until the live stream to explain it properly, but I'm pretty excited about the insights I expect people will be able to draw from it.

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

Earlier today, Have I been pwned (HIBP) appeared on a British TV show called The Martin Lewis Money Show. A producer had contacted me about this last week:

I Just wanted to get in contact to let you know we're featuring 'have I been pwned?' on the programme next week (Monday 28 Nov, 8pm, ITV) saying it's a good way to check if your data has been compromised. I thought it best to let you know in case you need to put extra resources onto it, we do have a tendency to crash websites with traffic!

I get this a bit - people saying the site will be featured or that they'll be hitting the API a lot or something to the effect that I need to prepare myself. I almost always barely see a blip, but just to be safe I scaled up from my normal S2 website instance on Azure to an S3 with double the capacity. I put a note in my calendar and kept an eye on my analytics when the show went out. I initially saw dozens of referrals from moneysavingexpert.com and assumed it was the same old story: "big" traffic is relative. Then, within about 60 seconds, the simultaneous users on the site went from about 200 to this:

That's about double the highest volume I'd ever seen in the past. Here's some brief points about how the service handled it and what I learned from the event.

Autoscale can only scale so fast

I use Azure's autoscale feature which is configured like this:

Get too much traffic and HIBP scales out with more instances of the same web server. Thing is though, this isn't "point in time" insofar as that 80% CPU utilisation needs to be sustained for a period of time:

If the duration is too short you get yo-yo'ing where you continually have instances being added too quickly then taken away because there's one or two process-intensive requests running. Scaling too early costs money; I'm paying by the minute with Azure websites. All this works great for organic increases in traffic to even very large levels, but not so well for sudden, massive spikes like this:

The only reason that graph comes back down quickly is that I manually added another 4 instances when I realised what was going on.

Traffic loss is inevitable

This is the harsh reality: if you want to use autoscale and you don't want to be perpetually over-provisioned, there will be times like this where traffic will be lost. Here's what happened earlier today:

Fortunately, there's only a few minutes there where there was any loss at all, but it still happened and that's a bit disappointing. But it's also the reality of unexpected success and I'm ok with that. Often people will have solutions along the lines of "well if you'd used this service / framework / pattern it would have been fine", but the simple reality is that whilst you're dependent on underlying infrastructure that's capacity bound, this will happen. The "serverless" model such as Azure's functions I've written about in the past promises scale beyond that so I'm pretty keen to see the promise of that come to reality in the future.

Sudden good traffic can look like sudden bad traffic

Part of the problem too is that from a pure traffic volume perspective, it can be difficult to differentiate wild success from malice. A little later on in the day I saw this:

This is malicious traffic and although not quite at the volumes of the good traffic earlier on, that pattern is remarkably similar in that we go from a steady state to a massively higher volume in an instant. The only reason it comes back down again so quickly is because my Azure Functions implementation blocks it at Cloudflare. But you can see the challenge, I'm sure.

Offloading traffic to Cloudflare helped massively

Take that earlier graph with the server errors then compare it to this one from Cloudflare:

Yes, I lost some traffic, but it could have been far, far worse. Cloudflare actually absorbed 76% of 1.3 million requests in a 15 minute period. Think about that - I could have had 4 times as much traffic hit my site during that period without Cloudflare and that would have had a profound impact on the number of requests that were dropped.

One of the reasons they're caching so much is that I configure page rules as follows:

That last one, for example, caches the homepage so a huge number of requests to the root not only don't hit my origin website, but are served up from one of more than 100 "edge nodes" Cloudflare has around the world so people get it super-fast too. That doesn't mean that every request comes from cache, they still occasionally pull content form the origin and if it goes down then you can still have problems.

You get this for free on Cloudflare and that's a really important point to make. This is a case where using their service without spending a dollar actually saves you significant amounts of money because that's a lot less infrastructure you need on the back end. I saved a bunch in bandwidth too:

That's 20GB of cached bandwidth in 12 hours right there which is small in terms of dollar terms, but sustain large amounts of traffic and the cost of that can start adding up.

Don't serve what you don't have to

A long time ago now after a very early experience of high traffic volumes, I realised how valuable public CDNs are for serving libraries like jQuery and Bootstrap. In fact, back then I lamented that I'd served 15GB of jQuery alone. It's not just the bandwidth costs of doing this, it's the fact that under heavy load situations, while the server is returning those JS files it's not able to do other things.

Use public CDNs like Cloudflare's to serve this content. If you're worried about tampering of the files then use subresource integrity attributes as well. It's all just part of offloading everything you don't need to do.

Measure the experience users receive

It's easy to get very focused on how many requests were served versus dropped in scenarios like this. But of the ones that were served, how fast were they served? And how did that impact the user experience? That's why stats like this from New Relic are great:

I could still be serving requests but if I see the response times getting up near a second then I know I've got issues. It's a similar deal with their apdex score:

This shows user satisfaction dropping as everything slows down.

Get told as soon as things go wrong

Alerting is absolutely critical and the first inklings I had that something was up came via Twitter:

@troyhunt more coverage #UkMoneySaving pic.twitter.com/zwjkNFfo6j

— GaryMcAllister (@GaryMcAllister) November 28, 2016

As good as Twitter is for early warning, it's not particularly reliable! New Relic was the first to let me know something was up:

Thing is, alerts like this are always sent after stuff is going wrong! I have alerts that tell me when things get busy but that only helps avoid outages when ramp up is gradual and you can respond in time, not when tens of thousands of people simultaneously land on the site. The alerts are valuable, no - essential - but in cases like this, that's mostly to help you fix broken stuff.

What I can do better

I'd love to break out the website itself and the API back end into fully autonomous units. Because they all run on the one site at present, when things are overloaded then everything goes down. I'd love to use Azure's API Management service but unfortunately at my scale, it's just not financially feasible for a free service like HIBP.

I'd also love to geo-distribute the service which is easy to do with Azure Traffic Manager. This would ensure that big traffic from, say, the UK doesn't bring down traffic from the US. But it's a cost issue again because obviously I'm now running multiple instances of the infrastructure.

But more than anything - and this is what's bugged me throughout the day - is when someone says "we're going to send you a lot of traffic", overcompensate then scale it back later. And it would have been so cheap to do too...

Summary

I could have easily handled this traffic spike and the bit that really frustrates me is that I could have done it for 40c. A mere $0.40 is what it costs to run an S3 web app service for an hour on Azure and looking back at the traffic, that's all it would have taken. Probably less actually; I would have only needed it for about half an hour!

But it's easy to play the "how much is enough" game in retrospect, much harder to do it when planning ahead. The traffic could have been triple what it was, it could have been nothing, it could come totally out of the blue and certainly that's happened before too. If I was running this as a commercial operation then it would be easy - I'd just over-provision all of the time. Running it on a shoestring is a different story though as I'm always trying to get as close to optimal as possible and that means that sometimes, stuff like this is going to happen. But because it's primarily a free service it also doesn't really matter that much that I lost traffic, it's just the perfectionist in me that's unhappy.

Oh, and for people that are interested in what the coverage was that caused all this, the show is now available online (about the 8 minute mark). You need to be in the UK though, if only there was a way for overseas visitors to "be" somewhere else...

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

A bit of a quieter week this time blog wise, but a very busy week in terms of HIBP traffic. It went pretty nuts on Tuesday with a spike the scale I'd never seen before which made things, well, "interesting". I also put the word out about an "ask me anything" live stream event I'm going to do early next week which should be a lot of fun. Oh - and the Indian pathology results exposed to the world - that's unfolding as I write this but the position from the lab exposing things like patient HIV results to the world right now is "we'll get around to it in Jan". The latest is that BuzzFeed has just written about it so go there to read the details in full and marvel at the quotes from those involved...

iTunes podcast | Google Play Music podcast | RSS podcast

1. Ask me anything about HIBP! (this'll be live streamed and I'll show people everything I can while I answer questions)
2. The mother of all organic traffic spikes on HIBP (so far..) (from 200 users on the site to more than 12k in less than a minute - fun!)
3. An Indian pathology lab has left 43k test results exposed on an American server (oh - and they don't want to fix it until Jan!
4. I've got a bunch of other events coming up (I'm also trying to get commitments for an April trip to pick up the excess demand from next month's)

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

I'm used to seeing large amounts of personal data left inadvertently exposed to the web. Recently, the Red Cross Blood Service down here left a huge amount of data exposed (well, at least the company doing their tech things did). Shortly afterwards, the global recruitment company Michael Page also lost a heap (also due to a partner, Capgemini). Both cases were obviously extremely embarrassing for the companies involved and they did exactly what you'd expect them to do once they found out about it - they pulled the data offline as fast as humanly possible.

And this is how it generally goes with incidents like this; lots of embarrassment, lots of scrambling to fix then lots of apologising afterwards. Which makes the behaviour of Health Solutions in India all the more confounding. Here's how it all unfolded:

On Wednesday, someone popped up on the Twitters and shared a link with me via DM which went to www.hsppl.com/pathology/downloads/downloadReports and returned this page:

You won't find anything there anymore because they've (eventually) been removed. But what you're looking at in that screen grab is files displayed in a folder with directory listing enabled, precisely the same vulnerable configuration the brought the Red Cross and Michael Page undone. There were 43,203 files in total relating to pathology reports because that's what Health Solutions does:

Now this is very sensitive data so let's take just a moment to reflect on the ethics involved when you discover (or are told of) an incident like this. It was quite clear what was in this folder so I didn't click on any of the documents. Not only did I not want to breach the confidentiality of those involved, I also didn't want to contaminate any logs which may later be subject to forensic review. Having been down this path many times before, one of the things (responsible) companies do after learning of an incident like this is trawl through the logs to get a sense of how broad the exposure is. It's not that I didn't just want my IP in there, I wouldn't want to hit it via VPN or Tor either because that only adds noise to the logs and makes it harder to get to the bottom of what's actually happened.

Having said that, there's another way to get a sense of what's in the documents without actually loading them and that's to see if Google has picked them up. Which it had:

It wasn't all 43,203 documents indexed, but then a bunch of them were .rar files anyway so that's to be expected. The problem was that not only were the files themselves indexed, the contents had also been cached. In other words, Google had been reading the pathology reports and the contents had now spread beyond the source web server.

Keeping the privacy aspect of things in mind, as unfortunate as it is that Google had indexed things it actually made it a lot easier to work out the scope of the leak without actually looking at the files. For example, one of the most sensitive pathology reports I would expect to find are tests for HIV:

So not only had the files been indexed by Google, they were available via cache and contained some very sensitive information too. This is about as bad as it gets in terms of exposed health data.

The next step was to get a sense of who Health Solutions was because all I had at this stage was a domain name. A quick browse around the website showed they were in Mumbai in India:

I found an email address on the website and emailed them at 13:55 on Nov 30. The email immediately bounced back with "No Such User Here"

So I tried the contact us form:

24 hours later, nothing. I then tried the WHOIS admin contact which appears to be a bloke named Sunil working for Maurya Consultancy Services:

That bounced back with the same error as the earlier email:

So I started digging deeper into the site and one of the first things I did was look at where it was hosted which, to my surprise, turned out to be in the US:

I first wondered if I'd made an error somewhere or misinterpreted the results, so I took a look on Shodan as well:

The host name there is server.askmyindianfellows.com which just goes off to a default cPanel page. However, it reinforced the Indian connection and a WHOIS on that name joined all the dots:

I took a bit more of a look around the site to get a sense of what was going on and I think this one image summarises it best:

A marquee tag on a page with a hit counter (broken) and a copyright of 2011. It's the trifecta of "this is old and unloved"! I was rapidly losing confidence in the likelihood of finding anyone who cared, so I turned to Twitter:

I need a Mumbai local to call up the local pathology center leaking all their patient data and have them get in touch with me. Any takers?

— Troy Hunt (@troyhunt) December 1, 2016

A bunch of people responded, one of whom was Pranav Dixit, a BuzzFeed reporter. Pranav was excellent, immediately understanding the severity of the issue and getting in touch directly with Health Solutions. This was Thursday evening last week, more than 24 hours after I'd begun trying to raise someone there. I thought this would do it - the files will be gone in no time now - and then we got the most unfathomable responses from them...

Pranav later wrote these in a BuzzFeed piece I'll link to shortly so I'm comfortable sharing the following statements from Health Solutions he sent me via DM. He'd gotten the lab on the phone - this is the lab running the site that exposed the data - and they told him this:

We are not the doctors, we and our franchisees merely do the blood tests and maintaining doctor-patient confidentiality is not our problem

How is this even possible?! The relationship with the doctor is inconsequential because it's the lab that's leaking the data! This alone was unfathomable, but then they went even further:

we are moving to a new domain in january and retiring the existing website, so these problems will be fixed in jan

These messages were sent on the 1st of December so in other words, Health Solutions were saying they were going to leave the patient data exposed for at least another month and it wasn't really their problem anyway because confidentiality wasn't up to them. They had one more thing to say on the time frame:

but till then, we are not planning to do anything about this

We were both gobsmacked by this. How on earth can you leak this sort of data and just not care?! Look, it'd be one thing if there was a heap of engineering work to be done in order to secure the patient records, but where we were then it was simply a matter of removing the files or even better, just turning off the site until it was properly secured. Yes, that could have a business continuity impact but I'd never seen this stop an organisation from securing sensitive data that had been publicly exposed. Never.

The following day, the files were still there. It was Friday now and tens of thousands of pathology files remained publicly accessible and easily discoverable, especially given the Google index. I decided to start applying some social pressure, but I didn't want to name the lab as it'd take one simple Google search with an "inurl" and the documents would be found:

This is not going well, the path lab doesn't want to fix. Here's redacted details, Indian journos DM for details: https://t.co/G1DsIPKZ5f https://t.co/roBn8SZX4i

— Troy Hunt (@troyhunt) December 2, 2016

I'm used to seeing inadvertent exposure of data, but it gets fixed immediately once reported. "We'll do it in Jan" is unfathomably reckless.

— Troy Hunt (@troyhunt) December 2, 2016

I honestly didn't know how this would play out - if they're adamant that they're not going to fix the exposure and I can't publicly shame them by name, what's going to change? As it turns out, BuzzFeed solved that conundrum by publishing a piece titled The Medical Reports Of 43,000 People, Including HIV Patients, Were Accidentally Released Online shortly after my tweets. Obviously, my preference would have been to see the data secured first, but their position was that given the path lab didn't care, it was fair game. They also elected to print a redacted image showing one of the exposed reports (an HIV test) and shared other information derived from the exposed data such as the involvement of children:

Some included in the breach are as young as 17

BuzzFeed had managed to get in touch with Rodrigues Kustas who they refer to as "an administrator at Health Solutions" and he had some rather, uh, "enlightening" facts to share. For example, security incidents weren't exactly a new thing:

Health Solutions was moving to a new website in January because their current one had been “hacked” several times

In a subsequent interview with The Hindu, he was quoted as saying:

The data leak was six months ago and by now we already have a new server

In that same article, he talks about a "hack":

While the website has been hacked, none of the confidential information on health issue of any of our patients has been compromised

Let us be crystal clear about this - publishing files to a website without any access controls and in a path with directory listing enabled is not "hacking", it's incompetence. That's a really important distinction because the term "hack" shifts the blame to someone else when it should rest squarely on the shoulders of Rodrigues and co. And while we're here, saying that none of the info was compromised is blatantly wrong; BuzzFeed pulled HIV test results! Even if Health Solutions went back through the logs (which they may not even have), the fact that Google indexed it all and stored it in cache means that the files were copied outside their environment by a third party and they simply have no idea who has seen them.

In the BuzzFeed piece, he then goes on to shed some light on how such a shoddy development job may have come about:

the lab’s website was developed by a third-party developer that he described as a personal friend

Those of us who work in tech have seen it all before - someone's brother's uncle's mate's dog does some web development and they built the site. Rodrigues then apparently "refused to provide any more details" but he doesn't need to because it's all over the WHOIS records and hosting arrangement! The site was built by Maurya Consultancy Services, except there's not much there at the moment:

That site is running on the same IP address as all the exposed records were found on. Only 4 weeks ago, it looked like this:

That's the last recorded snapshot on archive.org and it appears that the site has been disabled (scrubbed?) around the time of this incident. Doing a reverse DNS lookup on the host name shows a raft of other sites running on the same IP address:

• amcoweigh.com
• aphali.com
• askmyindianfellows.com
• biglife4u.com
• cafefumo.com
• coloron.co.in
• coloroncare.com
• dashclinic.net
• directproductmark.biz
• futureenterprises.biz
• gretdhara.com
• homebulkdeal.com
• hsppl.com
• kantalaxmi.com
• mauryaindustries.com
• mcslinc.com
• mymoneystation.com
• mysvls.com
• nixonchemicals.com
• ourflame.biz
• sagardevelopers.com
• shreelandscaping.com
• smileservices.biz
• sribalajimedicare.com
• swaroopkart.com
• tarseelexports.com
• thethomsonandthomson.com
• thomsonandthomson.co.in
• trimsnbeyond.com
• vatanucoolengineers.com

At least they were running; every single one now shows the same "we're upgrading our servers" message. However, the internet archive can help fill in the gaps and a bit of searching shows everything from a pharmaceutical export business to a network marketing site ("happyness" is one of the things they do) to one which simply has directory listing enabled and a backup of the site in a zip file at the root (a familiar pattern by this time).

It's now pretty easy to join the dots on who's behind it:

A one-man show run by Sunil Maurya based in Mumbai building PHP, the language used for the Health Solutions site. I'd normally be reticent to name and shame in this way but the gravitas of the situation deserves it. An "MCA Fresher" per the by-line above is someone who has just completed their Masters of Computer Application. In fairness to Sunil, he may have had no idea what he was getting himself into and the blame really has to rest with Health Solutions for not pausing to consider that maybe a personal friend fresh out of college wasn't the most responsible choice for this class of data.

Just to make things even worse again, a bit more Googling revealed this:

For the non-technical readers, this a "shell" which allows an attacker to remotely run commands on a server. Someone has been able to place the malicious software on the Health Solutions websites which then enabled them to perform a whole raft of other nefarious activities on the machine. Google indexed it on the 10th of November, less than 3 weeks before the exposed data was reported to me. It appears to be unrelated to the fact that the site didn't secure the pathology reports in the first place, but it speaks to how poorly managed the entire thing was.

Summing it all up, here's what we know:

Health Solutions hired a personal friend fresh out of school working for himself to manage tens of thousands of pathology reports on his own server he hosted in a foreign country along with a bunch of other unrelated sites that was infected with a malicious shell

This needs calling out because it's such a grave violation of trust on behalf of those impacted. But there's an angle to all of this which we can't ignore, an "elephant in the room", if you like, and it's this:

@troyhunt privacy of personal data, is still not a thing most people in India are bothered about.

— Anon desi (@AnonDesi) December 2, 2016

@troyhunt Privacy and security is a myth in India. Nobody gives a damn.

— Binoy Xavier Joy ⚡️ (@binoyxj) December 2, 2016

Now I'm not sure it's entirely that black and white because there are inevitably many people that do care, but there are also a different set of priorities in India. I spent a great deal of the last decade and a half building medical systems used across Asia Pacific and as I recently wrote, the development was often off-shored to India. One of the things I quickly learned is that for emerging markets in general, they have issues that far and away trump the privacy situation outlined here. Issues such as poverty and rapid urbanisation. Issues of low literacy rates and high infant mortality. These are foreign concepts to most people living comfortable western lives, but you can no doubt see how privacy of data such as this could be considered a "luxury" we enjoy by virtue of being in more developed nations. None of that should excuse this situation, but it hopefully helps explain why it wasn't approached with the urgency we'd normally expect to see.

Eventually, the data was indeed removed. I first saw it gone Saturday morning then by Sunday, the results in Google were gone too hence my publishing this piece now. Many Indians who've contacted me have expressed concerns not just about this issue, but what it heralds as they rapidly digitise services without yet having the privacy frameworks required to protect data (i.e. there's no HIPAA equivalent). And they're right - it's worrying - and inevitably what we've seen here is far more common than we know.

I doubt that Health Solutions will now contact the thousands of people impacted by this as we'd see mandated in other parts of the world. I also doubt there'll be any legal or regulatory recourse as a result of their incompetence. But what I do know is that what we've seen here is consistent with so many of the other incidents we've seen around the world in terms of the technical failings. I do hope India can get the regulations in place to hold people accountable when it happens again because with the rush to digitise this sort of thing, it will happen again.

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

I get a lot of requests from people for data from Have I been pwned (HIBP) that they can analyse. Now obviously, there are a bunch of people up to no good requesting the data but equally, there are many others who just want to run statistics. Regardless, the answer has always been "no", I'm not going to redistribute data to you. In fact, the requests were happening so frequently that I even wrote the blog post No, I cannot share data breaches with you.

However, as part of HIBP's 3rd birthday celebrations, I am going to share data with you, quite a lot of it. In fact, I'm opening up almost all the data in HIBP with a few very important caveats:

1. All personally identifiable information has been removed
2. All information about the domain each account is on has been removed
3. All sensitive data breaches have been removed

As much as I want to provide data for analysis, I don't want to put anyone at any further risk which is why the personally identifiable data is gone. I've been careful to ensure the system is not open for abuse by virtue of efforts such as the API rate limit and serving up all the raw data in one big file would obviously not further that objective in any way.

Removing the domain data means that insights about who's been impacted where can't be implied. I don't want someone working out precisely which services a company's staff has accounts on. Obfuscating the domain would still pose a risk: if you can work out what just one account on an obfuscated domain is (say because of the unique set of breaches they appear in when querying the live system), you could resolve the domain.

Removing sensitive breaches ensures that creative people can't find where an account was compromised in a breach I deliberately keep from being publicly searchable. For example, I myself am in 8 non-sensitive breaches and if you took those 8, found a row in the data I'm opening up and saw that those very specific 8 and an adult website were in the same row then that poses a privacy risk. Sensitive breaches account for 11% of the total data in the system so in the overall scheme of things, it's a small loss.

Let me talk about what I've actually done here. All the data about accounts in breaches is stored in Azure Table Storage. I've written at length about Table Storage before and how I designed the partitioning. It means that a single account in there looks like this:

{"Partition":"hotmail.com","Key":"troyhunt","Websites":"000webhost;Adobe;Dropbox;GeekedIn;LinkedIn;Patreon;Plex;Tumblr","Timestamp":"2016-11-16T08:36:51.1806398Z"}

That's obviously my record and you'll see that the domain of my email address is the partition key, the alias is the row key then there's a semicolon delimited array of pwned sites and a time stamp of when the record was last modified. What I'm doing is extracting just the websites for each account, removing any sensitive ones then representing it all in one line as follows:

000webhost;Adobe;Dropbox;GeekedIn;LinkedIn;Patreon;Plex;Tumblr

There are exactly 1,431,112,732 rows adhering to this pattern in the file I'm giving everyone today. That's a 15.3GB file which I could have distributed out just like this, but that would mean massive redundancy because you get a bunch of rows that look just like this:

LinkedIn
LinkedIn
LinkedIn
LinkedIn
LinkedIn

So what I've done instead is aggregated the results, grouping them by the impacted sites and putting a count next to them which brings it down to a 135MB text file. This means that there's only one row that shows only LinkedIn and it looks like this:

LinkedIn 105645374

In other words, there are 105M email addresses that appear only against LinkedIn but rather than using that many rows in a text file to describe it, there's just the one row. Other times you'll see rows like this:

Adobe;Aipai;CivilOnline;NetEase;NexusMods 20

There are 20 people that have been pwned in that unique combination of 5 websites. Of course there are many, many combinations with only 1 entry. My own, for example, is the only email address which appears in those 8 unique sites. (This should also help explain why I needed to remove the sensitive data breaches in order the protect privacy.)

Let me summarise exactly how these numbers break down as I know people who analyse data in depth like to understand these things precisely:

1. 1,989,141,353 - the number of accounts currently represented as being in HIBP. This is every occurrence of an account in a breached system so my email address has 8 records in there. This number also includes usernames; in cases like Snapchat, there are 4.6M records and none of them are email addresses.
2. 1,574,694,164 - the number of unique email addresses. It's 400M lower than the previous number because a bunch of email addresses have appeared in multiple data breaches and it doesn't include any usernames. (Sidenote: I don't tend to load usernames where email addresses are available instead.)
3. 1,431,112,732 - the number of unique email addresses which contains one or more accounts that are not sensitive. This means that almost 144M email addresses only appeared in a sensitive breach. (Incidentally, there are a total of 221M accounts in breaches marked as sensitive in HIBP.)
4. 2,399,307 - the unique number of website combinations accounts have appeared in. This is how many rows are in the file I'm sharing and there's a count against each one showing how many times this combination of sites appears against an email address.

For each row, you can then take the breach names and reconcile them to the list of breaches exposed in the API. What this means is that you can access all the other attributes of the incident, for example here's Dropbox:

{  
   "Title":"Dropbox",
   "Name":"Dropbox",
   "Domain":"dropbox.com",
   "BreachDate":"2012-07-01",
   "AddedDate":"2016-08-31T00:19:19Z",
   "PwnCount":68648009,
   "Description":"In mid-2012, Dropbox suffered a data breach which exposed the stored credentials of tens of millions of their customers. In August 2016, they forced password resets for customers they believed may be at risk. A large volume of data totalling over 68 million records was subsequently traded online and included email addresses and salted hashes of passwords (half of them SHA1, half of them bcrypt).",
   "DataClasses":[  
      "Email addresses",
      "Passwords"
   ],
   "IsVerified":true,
   "IsSensitive":false,
   "IsActive":true,
   "IsRetired":false,
   "IsSpamList":false
}

The value in the file I'm distributing is the "Name" attribute you see above. It often differs from the "Title" attribute in that the former is a stable, alphanumeric value not intended for public display whilst the latter is a human-facing value that may change (i.e. if Dropbox gets popped again and I need to differentiate the two incidents). For example, the name "ModernBusinessSolutions" is different from the title "Modern Business Solutions". Just something to keep in mind if you're reconciling data back to the breach entities.

This should give those who are interested in analysing data breach patterns a heap of info to work with. Ideas that come to mind include the number of breaches accounts appear in, the data attributes exposed about them, their exposure over time and all sorts of other things I haven't even thought of. I'd love to see people turn this into some awesome visualisations; my mate John Bristowe did this neat dashboard in Microsoft's Power BI recently and that's just with data already available via the public API:

If you create visualisations or other insights into the data, do leave a comment below and share what you've done.

Now, how to get the data: in order to save me bandwidth costs and help it easily spread to those who want it, you can download the torrent file or grab the magnet link here:

magnet:?xt=urn:btih:97C585A73AE62A81E5A562237A1B33301F70C51D&dn=HIBP%20Consolidated%20and%20Anonymised%20Data.zip&tr=udp%3a%2f%2fexodus.desync.com%3a6969%2fannounce&tr=udp%3a%2f%2f9.rarbg.to%3a2780%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80&tr=udp%3a%2f%2fcoppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2fopen.demonii.com%3a1337%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2770%2fannounce&tr=udp%3a%2f%2f9.rarbg.me%3a2790%2fannounce&tr=udp%3a%2f%2fglotorrents.pw%3a6969%2fannounce

And for the extra cautious, the SHA1 hash of the zip file is:

31FE882F3F7C7917F1B0F2F04BCBF667B3E407DE

I suspect there'll be a large amount of interest in this so I'd like to ask anyone torrenting it to leave it seeding for a while too if they can, especially in the early days while it's distributed around. The goal is to make this data broadly available and enable people to do awesome things with it, so for that I need some community support.

If you have questions about the data, please leave them in the comments section below. Keep in mind that there may be - no, will be - discrepancies. If you retrieve the same data breach from elsewhere online and extract the accounts, they probably won't match exactly because the pattern matching for email addresses may differ slightly. If you crunched the numbers from the data I'm providing here they may not match exactly with what I've represented for each incident on HIBP as I've done multiple loads of some incidents. There'll be rows in the file which appear to have too many pwned sites or an odd collection of them due to fabricated email addresses (that test@test.com person has been really pwned!) In short, expect to see some inconsistencies but what I can say for sure is that once you "rehydrate" that data and add up the counts on each row you'll get to precisely 1,431,112,732 records.

One more thing: when you take a look in the zip you'll see a license.txt file. This is exactly the same license as on the API itself, that is it's a Creative Commons Attribution license. You can use the data to do whatever you'd like (including for commercial purposes), but just be clear about where it came from. This is all part of being transparent and particularly when it comes to a file full of data sourced from breaches, I want it to be crystal clear what it is and where it's from so that it's not misrepresented.

Please take this data and do awesome things with it. If you find it useful and want to contribute back to the project, check out the donations page. Regardless, do share any insights you've gained in the comments below, I'd love to see what people can do with this!

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

Content security policies (CSPs) can be both a blessing and a curse. A blessing because they can do neat stuff like my recent piece on upgrading insecure requests yet a curse because they can also do screwy things like break your site. Now in fairness, the breaking bit linked to there was more because of Safari's screwy implementation than because of the CSP spec itself, but that brings me to today's post on yet another screwy browser implementation of CSP. This time, it's Chrome's turn and it didn't just cause content to be blocked, it actually cost me money. Let me explain.

I have a donate page on Have I been pwned (HIBP). I honestly didn't expect people to give me money for something I provide for free, but it turns out plenty of people are happy doing it and obviously, I'm happy when they do! A little while back, I had a few messages from people saying "Hey, your donate page doesn't work" to which I gave the tried and tested standard response of "Works on my machine". But it was more than that because I was seeing donations from other people come in so if something was broken, it must have been a real edge case. It wasn't until someone actually showed me their console output in Firefox that the penny dropped. Here's what happened:

Let's say you decide to shout me some beer because running this service is thirsty work:

You hit the "Donate" button then leave me a friendly error message:

So far, so good. Now you hit the "donate now via PayPal" button which gives you this:

This is all fine, now let's try it in Firefox:

Then hit the same button and... here's what you see:

Huh. You must not have clicked it, I'm sure if you click it again it will be just fine:

Alright, this is weird, let's check the console:

This is talking about a form action, so let's look at what's happening on the page form wise:

At first glance, I blamed Firefox. I thought I'd properly added PayPal to the form action directive therefore Firefox was to blame, right? On a whim, I pinged my mate Scott Helme who's done many, many cool things with CSPs in the past. And then we actually took a look at the source CSP:

content-security-policy:default-src 'self' https://www.google.com;script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com https://www.gstatic.com https://apis.google.com https://www.google-analytics.com https://cdnjs.cloudflare.com https://js-agent.newrelic.com https://bam.nr-data.net;object-src 'none';style-src 'self' 'unsafe-inline' https://cdnjs.cloudflare.com;img-src 'self' https://www.google.com https://www.google-analytics.com https://ssl.gstatic.com;media-src 'none';font-src 'self' https://cdnjs.cloudflare.com;child-src 'self' https://www.google.com;form-action 'self' https://accounts.google.com;frame-ancestors 'none';report-uri https://haveibeenpwned.report-uri.io/r/default/csp/enforce

Here's the important bit:

form-action 'self' https://accounts.google.com;

Where's PayPal? And if there's no PayPal, how come Chrome is able to submit the form? Because Chrome is broken, that's why. I added the form-action directive on the 25th of May and the penny didn't drop on what was happening until the 16th of July so in other words, I'd just gone more than 7 weeks breaking donations from any browser that wasn't Chrome and that actually recognises the form-action directive (so Internet Explorer, was, uh, "fine"...)

Scott reached out to the Chrome folks and submitted a bug report on July 21. Last week, it was finally fixed in version 55.0.2883.75 which includes this:

[$N/A][630332] Low CVE-2016-5225: CSP bypass in Blink. Credit to Scott Helme (@Scott_Helme, scotthelme.co.uk)

Which is great, because it means that Chrome now breaks as expected:

Now let's be clear about this - I screwed up. I should have added PayPal to the CSP and I didn't. I also didn't monitor my CSP reports which are submitted to Scott's free CSP reporting service, report-uri.io. Regardless of my own shortcomings on this, I wanted to write about both the ease with which you can screw these things up and the continual idiosyncrasies of browsers in implementing the spec. Which brings us to this:

Be very careful with CSP! You can easily break your things without realising it and you need comprehensive testing and monitoring to avoid problems.

On the monitoring front, part of the problem with CSPs is that the reporting is very noisy. I just took a quick look at my reports in report-uri.io and saw the usual plethora of things that require no action on my part: blocked URIs of "data", plugins attempting to load content from non-white-listed URIs and requests from adware-riddled machines.

CSPs are awesome, but exercise extreme caution when using them. The whole purpose of a CSP is to stop requests being made under certain conditions and merely adhering to the spec won't ensure that doesn't happen when it shouldn't. If you'd like to know more about CSPs, check out my Pluralsight course on browser security headers.

Sponsored by: Terbium Labs — Try Matchlight for free. Fully automated, full private Dark Web Data Intelligence.

This was a pretty jam-packed week which kicked off with the crazy, crazy Indian pathology data leak. You'll sense my frustration with the whole thing and frankly, I still can't quite get over it. Be that as it may, stuff like this provides us with endless material that speaks to how badly wrong it can all go with any data that gets digitised. There's that and a bunch of HIBP bits in relation to the AMA I did earlier this week and the 1.4 billion records I made available for analysis. All that and more this week!

iTunes podcast | Google Play Music podcast | RSS podcast

References

1. Pathology data spilled all over the place in India (down syndrome tests, HIV tests and more - all public)
2. Reused credentials used on your site - are you responsible? (this is an interesting discussion in terms of how you can defend against this sort of thing)
3. HIBP turned 3 so I answered a bunch of questions (the recording of the live stream is now on the blog from last week)
4. Oh - I also gave away 1.4 billion records (they're de-identified and don't include sensitive breaches)
5. I broke my CSP (again) so many people couldn't donate to me (I want content security policies to be good, I really do, but they're still super risky)

Sponsored by: Do you desire peace of mind? The hackers don't wait, secure your website and mobile apps with Gold Security today.

Many people will disagree with this post, not so much because it's flat out wrong but because there are so many different approaches one can take. It's a very subjective realm but I'm going to put forward some suggestions, make some considered arguments and leave it at that.

The context is twofold as suggested by the title: Firstly, I get a lot of people asking me about how to get a start in the security industry. I've regularly reverted with "stay tuned, I'm writing something" and this blog post is it.

Secondly, over most of last year and the first half of this one, I've been creating material to help people who want to pursue security careers. It's the Ethical Hacking series on Pluralsight that sets people up to get their Certified Ethical Hacking cert (CEH). I want to talk candidly about CEH - both the good and the bad - and give people a path forward that can get them started in the industry.

Here's my experiences, what I think makes sense today and most importantly, here's the tools to make a start.

How I got my "start"

I want to begin here because many people ask me precisely this - how did I get started in security. In my case, it was entirely by accident, but let me explain anyway.

I've (almost) always been a web developer. I started writing software in '95 whilst at university and from day one, it was software for the web. There were a few years before that where I made pocket money working in various part time PC support roles but for the most part, it's always been about building software. I spent a decade and a half doing that before I really began to think seriously about specialising in security.

It was a very organic shift for me and it all began with this blog. In fact, I can tie it back to this exact blog post in 2010 where I began writing the OWASP Top 10 for .NET Developers series. I wanted to write this because I saw a need: developers had a very poor grasp on security and I wanted to articulate it in a way they could easily consume. That blog series lead to my first MVP award a year later which lead to the start of speaking at events which lead to Pluralsight which led to, well, all sorts of things.

All of this ultimately led to where I found myself earlier last year which I explain in How I optimised my life to make my job redundant. As I wrote more and spoke more and ran more workshops, the opportunities piled up. It was never planned - I never sat down and said "I think I'll have a career in security" - and all that makes my path here a bit unique.

I'm not going to recommend that anyone take the same path because it was such an organic thing that simply responded to the environment around me. Other people are in different environments and doing the same thing won't necessarily yield the same results. To be quite honest, the more I think about my journey the more I feel it's been because of the way I communicate and explain concepts than it is my knowledge of security specifically. That's probably a topic for another day though.

Be that as it may, let's look forward and talk about careers in the industry for folks who are perhaps able to plan it a little better than I did!

Infosec is hot

This is a great industry to be in and the timing has never been better than right now. By virtue of you being here reading this in the first place you're inevitably tuned in enough to see just how much security features in the news each day. To my mind, there's a perfect storm of factors contributing to this including the availability of cloud services (it's cheaper and easier than ever to stand things up), the rise and rise of IoT (more devices collecting more data about more things), the actions of Assange, Manning and Snowden (love 'em or hate 'em, they've had a profound impact on our views of online security and privacy), the rise of hacktivism (many of us have had our data leaked under this banner) and many, many other factors. The whole role of infosec in national security is another massive consideration at the moment but again, that's probably not telling you anything new.

There's been many quantitative pieces on the value of security jobs in recent times as well. Earlier this year Forbes talked about one million jobs openings in 2016 alone, 209k of those just in the US. In a report from last year on the state of job market, security roles were found to have "grown three times as fast as openings for IT jobs overall":

That's also reflected in how well rewarded security pros are; high demand with low supply means a premium salary package:

That’s bad for employers but good news for cybersecurity workers, who can command an average salary premium of nearly $6,500 per year, or 9% more than other IT workers.

I looked around for other figures to further illustrate the point and there's bucket loads of them but frankly, it all contributes to the resounding chorus of "security is hot, there's a heap of demand and it pays a lot of money". Let's get onto carving out a career in it.

Online identities are still a smart career move

I want to start here because I firmly believe that identity and personal brand are extremely valuable assets, regardless of where you choose to specialise. My first ever blog post was about why online identities are smart career moves and I wrote it whilst having a very diminutive presence compared to today. I didn't know how true that blog post would turn out to be and it's one of the first things I recommend people do these days.

Answer questions in places like the Security Stack Exchange site. If you don't know the answer, go and find it then answer it with references. If you have questions you don't know the answer to, ask them there. Just do something to start creating a presence.

Put code on GitHub. Make pull requests to other people's code. Branch interesting stuff. Get involved because all of these things lead to interactions with other people which leads to relationships which leads to opportunities.

Actively go out and create those relationships with others in the industry. Engage with them on Twitter, comment on their blog posts, meet them at user groups. Most people who've successfully made their way in whatever industry it is they've decided to specialise in happily answer questions when asked or take feedback on the things they've created. It gives you an opportunity to make connections.

All of this is free and you can start doing it right now. However, none of this is going to land you a job tomorrow; it will certainly improve your marketability over time, but it needs to be augmented with tangible skills which brings me to the next point.

Certifications and Ethical Hacking

There are many good dedicated security certifications out there that require various levels of investment in both time and money. For example, CISSP (Certified Information Systems Security Professional) is very popular among many security professionals. OSCP (Offensive Security Certified Professional) is another and by all accounts, it's amazingly in-depth and gruelling.

CEH (Certified Ethical Hacker) is the one I've been working on with Pluralsight and I want to start by explaining the background on how I got involved, not having previously done anything CEH related in the past. As I wrote about when I launched the first course in the series, Ethical Hacking content was the number one requested material on Pluralsight:

I was over in Salt Lake City in Feb last year for their author summit and they very enthusiastically approached me about writing some of what would ultimately become the 21 courses required to meet the CEH syllabus. I was initially reluctant as I knew very little about the cert and most of what I knew about CEH was from when they had their website defaced the year before! However, as I looked into it a few things began to resonate.

For one, the CEH certification is very broadly recognised and in some cases, required for certain roles. For example, the DoD expects to see it for all information assurance positions:

The United States of America Department of Defense issued Directive 8570 in 2004 to mandate baseline certifications for all Information Assurance “IA” positions. In February of 2010, this directive was enhanced to include the Certified Ethical Hacker across the Computer Network Defense Categories “CND”.

A trawl through the job ads will show many positions looking for CEH qualified individuals and certainly EC Council has made it a very well-recognised cert. I've heard this levelled as a criticism - that they've marketed the brand very well but the content lacks depth - which brings me to the next point.

When I looked at the syllabus of those 21 courses, there were a bunch of them which I thought would create really good standalone courses on Pluralsight regardless of their alignment to CEH. Further to that, many of them were in areas which I'd spent a lot of time teaching and writing about over the years. In fact, the very first course I ultimately created for the Ethical Hacking series was on injection, the same topic that first OWASP blog post had been on 5 years earlier. As I read through the syllabus, the idea of creating content on a number of the modules started to make a lot more sense. Pluralsight also recruited Dale Meredith whose been a Certified EC-Council Instructor for the last decade. He brought knowledge of the cert with him and tackled most of the non-web-centric content. Between the two of us we ultimately built 18 of the 21 courses, each working to our respective strengths (the other 3 courses were created by Jason Helmick and James Murray).

Ultimately, it boiled down to this: The content was very highly requested, the CEH cert very broadly recognised and there was a good chunk of the content I was already specialising in and had taught many time before.

The next issue was around how deep the material went and this is where I wanted to do things a bit differently. Dale and I discussed that at length many times with the folks from Pluralsight as well and I really wanted the content to go deeper than the reference material we were looking at from EC Council. To my mind, the content was very tool-centric so it would talk about using particular software products to mount attacks such as SQL injection. I really wanted people to understand the mechanics at a lower level so when I wrote the course I began with talking about the way queries and data are concatenated then went onto manual exploitation via parameter tampering then eventually, automated tools such as sqlmap. It meant that ultimately, that one course on injection was 5 and a half hours but IMHO, it was the right course to write. It's the SQL injection course I'd write whether CEH existed or not, which brings me to the next point.

One of the provisos of me doing the series was that I wanted every course to stand alone. For example, if someone is just interested in SQL injection and they go and watch that one course, they shouldn't miss out on anything by not doing the entire series. That's also why the courses talk about defensive practices; there's an entire module devoted to data validation, parameterisation, ORMs and so on and so forth because that's what a holistic course on the topic should do. This is the SQL injection course I wanted to write - they're all courses I ultimately wanted to write - which means this:

This is Pluralsight's most requested content ever. We've created the best courses we possibly could on discrete security topics and they're also a superset of the Ethical Hacking modules. This makes it easier than ever to skill-up on security in general and CEH content in particular.

I sum it up in this short interview about why I feel that technical depth is important:

"Be able to do and not just tell." — @troyhunt. Why we've launched a new security category: https://t.co/NWM3uNTHHM pic.twitter.com/4vN2YRaOPz

— Pluralsight (@pluralsight) July 27, 2016

So that's what we've created: over one and a half gruelling years with 8 courses from me, 10 from Dale and a few from the other guys with a combined total of 75 hours of viewing content. You can go through that material then sit your CEH exam with confidence and come out the other end of it as a Certified Ethical Hacker. The question people need to ask themselves though is if this cert makes sense for them and indeed if any cert does.

When a certification does (and does not) make sense

Certs are a means to an end. People get them because it helps them build knowledge and it helps them land a job. There are times when that makes sense and indeed there are times where it doesn't so let me give you an example of each.

If you're starting out in the industry and don't yet have experience to land you the job you're really after, a cert makes sense. In fact, even if you do have experience, a cert can be the thing that provides independent validation of your skills and sets you apart from the next candidate. It can also be a bargaining chip; demonstrated knowledge in the form of an independent assessment is another tick in the "strengths" column.

Where it makes less sense is when someone has that existing track record of demonstrated competency. It also makes less sense if they have the knowledge in these areas they want already; I don't see a lot of practical value in someone sitting their CEH just to get the title and continue doing the same thing in the same role (the exception here would be if that enables them to better negotiate their salary or working conditions).

The point is that you have to know what you're actually sitting the cert for. Further to that, it's important to recognise that this is a stepping stone in a career and indeed you could say the same about any cert or degree or other formal education. You don't do this, say "well that's that done", then settle in for a comfy life of fat pay packets. What any form of further education does is shows commitment; it shows you're serious enough about the discipline you've chosen that you're willing to invest the time, learn the content then demonstrate the knowledge, and that goes a long way with employers.

One last thing here and it's a bit of expectation management which was really clearly highlighted in a piece from last week about intensive coding schools. You're (almost certainly) not going to do any sort of education and with that alone come out and earn big bucks right away. Anyone that tells you elsewise probably has a motive that doesn't involve your best interests! A cert like CEH may open doors and create opportunities you wouldn't have had otherwise, but that's only part of the story.

Nothing beats experience!

This is why I wrote the section about online identities earlier on because that's a great way of demonstrating experience. Augment these things - do everything I wrote about there and do the cert thing and if you can go armed to a potential employer with both of those, you're starting to look pretty good.

Learning with Pluralsight

Lastly, if you've read this far and are thinking that maybe this might be the right path for you then let me talk a bit about Pluralsight. I obviously have a vested interest in people learning from them (I'm paid royalties based on course views), but this is my honest view on the best possible way to get started. I wouldn't have created the content if I wasn't fully endorsed in it.

What I like about Pluralsight's approach (and one of the reasons I started authoring for them 4 years ago), is that they're intent on making knowledge accessible to everyone. The commercial model for the licensing reflects this - you pay $29/m and you get access to everything. You can watch as much or as little as you like but for less than $1 a day, you can take courses such as those mentioned above then go and sit your CEH exam (note that EC Council still has a fee when you do the test).

Being online training, Pluralsight courses are self-paced. You can do them in the evenings, do them on the train on the way to work (you can take them offline as well) and indeed do them in the office as part of further education for your job. You can play them at 1.x speed (without me sounding like a chipmunk!) and you can pick and choose just the modules or clips you're actually interested in. Pluralsight has become enormously successful because this model of learning works so well for so many people.

There are obviously many different paths people take to arrive at their respective security jobs. Mine is totally different to what I've suggested here and as I mentioned earlier, I wouldn't exactly recommend that approach to anyone else. But as a means of getting started and doing so much so easily, I honestly can't think of a better way.

You can get started right now, Dale and I had a blast creating the content and we're enormously happy with the result!

Sponsored by: Do you desire peace of mind? The hackers don't wait, secure your website and mobile apps with Gold Security today.

Most readers here understand security fundamentals. They know what makes a strong password, what the padlock in the address bar above means, why software updates are important, the value of locking their mobile devices and some of dangers we face with the internet of things. But equally, most of our friends, relatives and significant others don't. We know this because we're continually doing tech support for them and we experience the horrors of their security profiles first hand!

Recently, Varonis asked if I could build a course for these folks, the ones that really need it. It's a change of pace to most of my courses that are targeted at technology professionals and obviously that means covering the fundamentals is the focus. We decided to focus on 5 internet security basics and the best bit is that they're giving it away for free. Not free as in "give us all your personal details first", but free as in it's on YouTube with no ads and you can watch it right here:

So for all my tech friends approaching the holidays and the inevitable influx of "hey, would you take a look at this for me" as they spend time with loved ones (and I've certainly seen some tech support requests challenge that description of family members...), sit them down in front of this for an hour and a bit. I hope it's as valuable to you as it is for the people that really need to understand it!

Sponsored by: Do you desire peace of mind? The hackers don't wait, secure your website and mobile apps with Gold Security today.

This week begins with the biggest of big breaches - the one that finally broke the big "B" - Yahoo (version 2). It's a massive story and I spent a lot of time yesterday answering media queries about hacker things related to data breaches. I talk about that at the start of this weekly update as well pursuing a career in security, providing an internet basics course for free via Varonis and how my blog on Ubiquiti network bits is still getting massive traction.

iTunes podcast | Google Play Music podcast | RSS podcast

References

1. The crazy, massive, huge Yahoo breach (there's a heap of angles to this, short interview with me there, longer set of thoughts in the weekly update video)
2. Here's my take on where to get started with a career in security (there are many, many, many different paths you can take, this is the one I reckon is a great first step)
3. Free course! Varonis and I teamed up to educate the masses (it's probably not for you, it's for you to get other people to watch)
4. My Ubiquiti post hit number 1 on Hacker News! (yes, I know I've mentioned it before, but this gear still rocks and heaps of people are jumping on it)

Sponsored by: Raygun — Discover, diagnose and understand issues in your codebase — Installs in minutes — Try free for 30 days!

Trust is a really difficult thing to define. Think about it in the web security context - how do you "trust" a site? Many people would argue that trust decisions are made on the familiarity you have with the brand, you know, brands like LinkedIn, Dropbox, Adobe... who've all had really serious data breaches. Others will look for the padlock in the address bar and imply by its presence that the site is trustworthy... without realising that it makes no guarantees about the security profile of the services sitting behind it. Then there's the security seals placed on the page and, well, just go and read clubbing seals if you're not already aware of just how fundamentally irrelevant (and even dangerous) they are.

But here's the thing: all of these go into that little processor in people's brains which helps them make a judgement decision on the trustworthiness of the site they're visiting. Rightly or wrongly, they matter.

The trick now is to look at which trust indicators actually make sense not just in the confidence they instil in people visiting a site, but in the actual security benefit they provide. For example, Ashley Madison choose to put a fake security award on their site which probably gave many people more confidence in them whilst setting out to have an illicit affair, but ultimately meant absolutely nothing. An extended validation certificate (EV cert), on the other hand, actually does mean something. I recently decided to get one for Have I been pwned (HIBP) and I want to take you through the process here.

Why EV?

Let's start with a definition:

An Extended Validation Certificate (EV) is a certificate used for HTTPS websites and software that proves the legal entity controlling the web site or software package. Obtaining an EV certificate requires verification of the requesting entity's identity by a certificate authority (CA). Web browsers show the verified legal identity prominently in their user interface, either before, or instead of, the domain name.

What this means in real terms is that when you look at HIBP now, you see this:

This states both HIBP's name and my own (I'll explain why later) as well as the country of origin. I had to jump through a lot of hoops to verify who I am and what HIBP is in order to get that cert. Compare that to the site you're on now:

When you think about HTTPS, you should be thinking about confidentiality (your data is protected from eavesdroppers), integrity (it can't be changed in transit) and authenticity (you know who you're talking to). When you're on my blog, you know you're on troyhunt.com but you have no idea who owns it. Now in my case you could go and do a WHOIS and get a pretty good idea, but there are many other cases where content is served over HTTPS and the actual entity controlling the site is difficult to determine. Particularly with the rise and rise of Let's Encrypt and Cloudflare, getting a padlock in the address bar has never been easier but this doesn't actually tell you anything about who's behind it.

This is an excellent summary from CertSimple who I'll come back to a little later:

For HIBP, trust is particularly important. It's running in a space with a lot of shady operators doing a lot of dodgy things and I regularly get the question "why should I enter my email address into some random site"? The EV cert doesn't provide any greater security or protect your data any better in transit (this is really important too - make sure you understand that) but it does give people greater confidence in who's behind the site. As you'll see shortly, it's not easy getting an EV cert and when you do see one like on HIBP, at least you now know who you're trusting your data with.

Who should you get your EV cert from?

Neither Let's Encrypt nor Cloudflare offer EV certs; you can't go along and get a freebie from either. However, what you can do is bring your own along to Cloudflare which is what I ultimately did. The question now is where to get it from.

Comodo is the world's largest provider of certs but the problem with them is that they're also not very nice. After the shenanigans they pulled with Let's Encrypt, I really didn't feel inclined to give them any money.

I had a chat to my mate Scott Helme about certificate authorities (CAs) who I could use and he recommended CertSimple based on him knowing Mike MacCana who founded the company. I later met up with Mike in London and he's a great bloke (and fellow Aussie) genuinely trying to do good things with the way CAs issue their EVs. He's trying to make the whole process both simple and fast:

That's personalised for my locale, you can go and use CertSimple to issue certs across the globe. Ultimately for me though, it wasn't simple nor fast although none of that was CertSimple's fault, in fact they helped enormously as I worked through the requirements. Let's get onto what you need because it's not as simple as anyone going out there just going and getting an EV cert for any domain they own.

What do you require to obtain an EV?

Frankly, this was the hardest part of the entire process for me and I had a false start along the way too. In a nutshell, you need a legal entity that can own the cert and this is where it all starts to get tricky, not least of which because the nature of legal entities differs around the world. Do proceed carefully and do your own research, I'm going to tell you about my experiences in Australia and they'll probably be different to other parts of the world.

In Australia, a legal entity that would be eligible to apply for an EV cert is typically either a proprietary limited company or a business. The former is a more expensive construct that requires annual returns submitted to ASIC (the Australian Securities and Investments Commission) that come along with a cost of A$249 each time. They also require annual tax returns and whilst that makes sense for an actively trading company, that's not what I needed for HIBP. I do actually have a proprietary limited company I do all my work under called "Superlative Enterprises Pty. Ltd.", but obviously that's not the name I wanted on the cert.

A business name, however, is a much simpler affair. You can register a name online and it costs a mere A$78 for 3 years. It gives you a name under which you can conduct business and little more. However, it also means you can use that name to register an EV cert as it does become a recognised business that's searchable online and is accompanied by a record of registration. So that's what I did - I set this up:

The summary of the business name and business name holder details then looked like this:

Which is about the time everything started to go wrong. You see, when you register a business name you need to provide an ABN (Australian Business Number) and that belongs to the entity that owns the name. In my innocence, I thought "I've got an ABN, I'll just use my existing company one" and that's why you see the business name holder above. The problem is though, that would mean the address bar would look like this:

I didn't want the company name in there, not because I wanted it hidden for any nefarious purpose, but because it just didn't make any sense. The whole point of the EV cert is to build trust by providing a name that people recognise and can look at and say "ah, I know who that is" and this just wasn't going to work. So I went back to the drawing board.

Actually, that was late September then in October I was in London and caught up with Mike who helped me work out a better way. When I got home again, I de-registered the business name and then registered it again:

Looks the same, right? Almost, let's look at those business name holder details again:

I simply registered a new ABN under my name and then re-registered the HIBP name under that. This meant that if the chain of ownership was followed it stopped at me which is why the very first screen grab of this post shows "Have I Been Pwned (Troy Hunt) [AU]". When I set out to do this, I hadn't expected to see my name there at all but frankly, I'm glad it's present. I'm very closely linked to the service and I think seeing both HIBP and my name next to each other are a good thing in terms of transparency and the confidence people should have when they see it.

An important footnote on all this: it was much harder for me than it would be a registered legal entity trading under its own name. For most people looking to acquire an EV cert they'd skip this entire section and jump straight to the next bit. My scenario is almost certainly an edge case; however, it was worth explaining so there's a better awareness about what goes into getting a cert of this nature.

So that's the back-story on the legal entity I needed, let's get onto actually obtaining the cert.

Acquiring a cert via CertSimple

Let me walk through the process of actually acquiring the EV cert which frankly, after all the business name mucking around, was the easy bit.

CertSimple is true to the name and it all kicks off here:

This obviously isn't exactly how my cert ended up looking in the address bar and that's some feedback I've fed to Mike. Whilst my case was a little unusual, a more common scenario would be registered business names then owned by a parent entity which may need to appear in the address bar. (Yes, I really do live in a place called Surfers Paradise and yes, it is!)

Next up, you'll need a private key:

Give it a bit of identity info:

And choose how long you want the cert for:

And here's the other thing with an EV cert - you pay a lot more money. This is now in USD and where you'd normally be looking at around $70 a year for a normal cert (those issued by Let's Encrypt and Cloudflare aside), you're talking more than 3 times that for an EV. The trust that this class of cert creates doesn't come free and when you consider the steps yet to come, you'll see why. I decided this was a worthwhile investment for HIBP because of all the reasons outlined earlier and ultimately, 65c a day is a justifiable amount to spend. I could have gone other places for the cert that might have saved some dollars, but with all the mucking around I'd already done I wanted the process to be easy and it was well worth the spend with Mike and co to get that.

Moving on, we're now into the validation process and this is where things start to get a bit more manual:

Verifying my email address was the easy bit:

We're seeing DigiCert here because they're ultimately the CA CertSimple uses to issue the cert. They're the ones who need to do the verification and bundle it up into a cert. They also do certs for the likes of GitHub, Facebook and the US Department of Homeland Security so it's good company to keep.

And now we reach the end of the automation chain:

DigiCert needs to validate both the business name authenticity and that I am who I say I am. The former is done through a combination of me submitting the registration documents and them doing publicly available searches via ASIC. The latter was done by having a Skype video call with me where the engineer asked me to hold up my driver's license to the camera whilst she snapped a pic of it under my face:

(Not included in photo: my driver's license!)

And that was pretty much it. There was a little confusion that ensued due to the old business name still being searchable whilst pending de-registration but we got through that and the cert promptly arrived:

From there it was the usual process of mucking around with certificate formats to get from what's provided by the CA to what's required by the host, but a bit of OpenSSL and that was sorted out pretty smartly. A few minutes later and you get the first screen grab of the post or if you're an Edge user, you get this:

Or for the iPhone folks:

And so on and so forth. It's particularly prominent in Safari on iOS as the entire URL disappears, replaced by the controlling entity's name (it also drops the country of origin).

And that's it - green bar, business name, my name, job done!

Summary

This whole EV cert thing is hard to measure in terms of value; I have no idea how many more people will put their email address into HIBP or how much more media or good will or donations it will get. No idea at all. It'll help a bit when there's impersonation in the same way as HIBP's verified Twitter account helps, that's honestly the most valuable situation I can conceive of right now.

Websites will go to quite some lengths to try and create a sense of trust and you'll see all sorts of stats as to why one company's seal increases conversions by [whatever]%. But what I do know is that it adds transparency and legitimacy to a realm that as I mentioned earlier, tends to be inhabited by a lot of shady characters and that's gotta count for something.

Sponsored by: Raygun — Discover, diagnose and understand issues in your codebase — Installs in minutes — Try free for 30 days!

The title says it all and the details are on their blog, but there's still a lot to talk about. Self-submission to HIBP is not a new thing (TruckersMP was the first back in April), but it's extremely unusual as here you have an organisation saying "we got hacked, we'd now like you to make that data searchable". This is in an era when most organisations are doing their utmost to downplay the significance of an event like this too.

This incident comes at a time when I'm writing up a fairly heft blog post on how organisations should communicate in the wake of a data breach. There's a lot of examples in there from previous incidents - mostly around what you shouldn't do - but I don't want to dwell on those here. Instead, I'd like to highlight some of things that really stand out to me in the way Ethereum has communicated this incident:

1. They communicated promptly: they learned of the incident on the 16th and had a blog out on the 19th
2. They were direct and honest: they disclosed precisely what data attributes had been compromised
3. They provided technical detail: there's info on hashing algorithms and a breakdown of what was used where
4. They explained how it happened: yes, there's limited info but this is one case where they need to be a bit selective about how much is shared
5. They've already reset passwords: this is important in terms of immediately mitigating risk
6. They explain what else they're doing to stop it from happening again: they need to rebuild confidence and this is an important part of that
7. They're apologetic: the post ends with "We deeply regret that this incident occurred"

That last point is enormously important; we're so used to seeing companies say things along the lines of "sophisticated illegal malicious criminal cyber-actor" or other words that blame the intruder rather than own the problem. Make no mistake - there was illegal activity that went on here that could have serious ramifications if the perpetrator is identified - but rather than trying to shift blame in that direction, Ethereum is owning the problem and acknowledging their own shortcomings. They talk about an "attacker" and "unauthorised access" but spare us the sensationalism we're so often confronted with.

Ethereum is working in an area where trust and transparency is paramount, indeed they're two key value propositions of blockchains. The way they've communicated this incident and their willingness to contribute the data to HIBP should tell you a lot about the ethics of the way they run their organisation.

In order to make the data searchable in HIBP, Ethereum sent over only the email addresses that had been compromised as I don't require any other data. They also provided a schema of the impacted database so I could properly document the data classes that were exposed.

There are now 16,431 email addresses from the Ethereum forum breach searchable in HIBP.

Sponsored by: Raygun — Discover, diagnose and understand issues in your codebase — Installs in minutes — Try free for 30 days!

I was shopping around for a new exhaust system for the car the other day and I found exactly what I wanted via a seller on Facebook. I really wanted to get some more specs on it though so I did what any normal person would do and Googled for it, finding a result titled "Boost Logic Nissan R35 GT-R 4" Titanium Exhaust" and linking through to a page on the official Boost Logic website. However...

Now this, clearly, isn't a good look. This is the official site and not a spoof or phishing site, yet Google had just put up a massive barrier to entry. It got me thinking about the old adage we hear so many times in security, the one that goes like this:

But we don't have anything of value on our site anyway

This defence is frequently preceded by an observation of a security deficiency somewhere and a suggestion that perhaps they're taking unnecessary risks. The opinion that many hold is that without the presence of credit cards or passwords or some other piece of useful data, the site simply doesn't pose any value to malicious parties.

This all got me thinking about the value of reputation and how it's exploited by malicious parties. I started looking around at some of the spam I was receiving (often bypassing spam filters), and saw the same patterns over and over again. For example, it was suggested that I may need to login to my Outlook account to fix some problems:

That's actually a pretty slick looking phishing page but the domain gives it away. Here's the actual site which as best I can tell from Google Translate, is a French Wordpress site about a kids choir event:

Look carefully though and you might notice the two images above show my VPN enabled. That's because when I tried to load it direct via my broadband connection at home, it did this:

This is reminiscent of the Boost Logic site earlier on where there's an upstream control blocking what has been identified as a malicious site. Oddly though, I can use the exact same telco but on my 4G connection and the page loads without being blocked.

And that's the other thing that's noteworthy with the way content is blocked - it's very inconsistent. For example, here's another pretty well put together Outlook phishing page:

Same deal as the earlier one in that it's an otherwise very innocent website behind it, except that this time both of them happily load over my home broadband connection:

But now here's a twist - I turn on my VPN (I use Freedome) and that's it - the site is now cactus:

You might have noticed I'm using the iPad for each of these tests as frankly, I've got a much greater degree of confidence that a heavily sandboxed iOS running without plugins is more resilient to nasty things. But let's give that last one a go in Chrome on the desktop anyway, just for a quick look:

And that's that site now pretty much dead in the world's most popular browser.

The same story plays out again and again and again. Another Outlook login page:

Which is just a London real estate site:

Outlook again (you seeing a theme here yet?):

Except it's not, it's actually a provider of security services (never a good look to have that sort of thing pwned!):

Even more Outlook:

And this page doesn't even have anything on it yet:

But this is really the point of the whole post and indeed the title of the page: the value your site has to attackers is not just the data, it's the reputation. It doesn't even need to be a good reputation in terms of it being a well-established site with lots of inbound links, it simply needs to be a site that doesn't have a bad one. That's the value to attackers - a launchpad that's not blocked by any of the mechanisms I showed above - and there's a very small opportunity to exploit that reputation before large swathes of people are blocked from the site altogether.

And just as one final adjunct to the story, this one happened just yesterday:

@troyhunt Either I have an incorrect setting or the guys at Norton are screwed up... I am certain it's not you. pic.twitter.com/UqjxO8wTOq

— Charles R. Smith (@softwarnet) December 21, 2016

To say I was surprised is an understatement! In this case, my blog was just fine but because I had one file on a totally different sub-domain that was classified as a "hacking tool", here we are. Fortunately everything came good as soon as I hid deleted it, but it just goes to show how fickle reputation can be.

So there it is: many different examples of why whether you know it or not, your site has something valuable to attackers, it just may not be what you originally thought it was.

Sponsored by: Raygun — Discover, diagnose and understand issues in your codebase — Installs in minutes — Try free for 30 days!

Almost done for the year and I've gone beach-style, if not in location then at least in attire. Xmas in Australia is all about the outdoors, the water and usually generous helpings of cold prawns so a little bit different to many places. But like everywhere else, the cyber things keep happening and there were a bunch of things on the agenda this week ranging from EV certs (largely a physiological discussion IMHO), to the Ethereum forum hack (or more specifically, how well they handled it) to how websites - any website - has something really valuable to attackers: reputation.

Thanks for the continued viewership and listenership folks, I hope everyone is getting some good Xmas downtime.

iTunes podcast | Google Play Music podcast | RSS podcast

References

1. Extended validation certs - good, bad or otherwise? (I reckon it's extremely hard to measure but does "some" good)
2. There's a lot we didn't know this year (the takeaway from 2016 was we learned how many breaches there'd been we never even knew about)
3. The Ethereum forum got hacked (but they did a really, really good job of communicating it then donated the data to HIBP)
4. Every website out there offers something of value to attackers - reputation (this is the rebuttal for every time someone says "but we don't have anything of interest to attackers on our website")

Sponsored by: Raygun — Improve UX performance with Raygun Pulse — Installs in minutes — Try free for 30 days!

I was preparing for an upcoming event the other day and very nonchalantly fired off a tweet whilst doing so:

As a conference speaker, about the most annoying thing you can ask me to do is to use your slide template...

— Troy Hunt (@troyhunt) December 16, 2016

Within short order, it somehow received hundreds of likes and retweets with many chiming in about the things that frustrated them about speaking at events. There was a lot shared that resonated with people and it struck me as odd, not least of which because almost every speaker at almost every tech conference contributes their time for free. This is news to many people - they think we're riding the speaker tour gravy train - and that makes it all the more unusual that conference organisers often make our lives harder than what they should be.

As the replies to that tweet rolled in, some patterns emerged that I thought I'd jot down here in the hope that they can be seen from the conference organiser's side and perhaps used to make life a little easier on everyone in the future. Here's the top 10:

1. Forcing a slide template on people

This is obviously the logical place to start and I want to explain exactly what the problem here is, particularly given a few people retorted with "a well-designed deck should be able to simply apply another template anyway".

A conference talk is about so much more than just words or pictures on a screen that you talk to. You're trying to create a mood and convey emotion via the imagery you use. In my case, this particular talk calls for a lot of dark themes and shady characters; augmenting this with fluorescent logos totally changes the feel of the talk. The time I'd put into refining a cohesive theme that matched the subject matter and complimented the talk felt somewhat wasted and that's a real shame not just because of how it now makes me feel about the talk, but because it'll detract from the audience experience too.

Adorning every slide with a conference logo or template does nothing for the attendees of the event. I mean they're not seeing it and saying "Oh yeah, I'm at the ABC Conference, I totally forgot until I saw that massive logo on the screen". Now I totally get asking presenters not to adorn every slide with their company logo or push their product ad nauseam because that experience sucks too, but they need to be provided with free artistic reign.

This is what we as both speakers and attendees want: we want to see lovingly-crafted visuals, we want to be told a story that we can get engrossed in and more than anything, we want to learn and be entertained. Standard conference slides do nothing to further any of those objectives, indeed they actually detract from it.

Oh - and if a conference requests specific fonts or slide software (i.e. not allowing the Mac folks to use Keynote), that's probably not going to go down well!

2. Asking for slides in advance

This is probably the most common complaint I saw in response to my earlier tweet. Some events request slides weeks or even months in advance and it makes things absolutely painful for a number of reasons.

One of those reasons is that in the tech industry, a lot can change even day by day. What makes sense today is not necessarily going to be accurate next month. Yes, there's usually a provision to submit revisions up until the talk itself, but it causes a heap of other problems too. This year I've probably done 50 separate talks, courses, webinars, user groups and other events I have to prepare for. Trying to tie things down way in advance of when they're actually needed whilst also trying to work on the things I actually need to do in the short term makes things enormously difficult on me. Frankly, I end up submitting something - anything - just to tick the box then heavily revising it closer to the date on my own schedule.

It also begs the questions "why"? What value does it really provide the event to have content so far in advance? Perhaps they want to review and assess for compliance to the standard template in which case, see the previous point. Maybe they want to censor or modify in other ways which worries me greatly because it implies that from looking at pictures or written words alone they can assume whether the talk is appropriate. The reason that worries me is because it's such a tiny part of the overall experience and without seeing it in the context of the narrative that goes alongside it, it's very easy to take them out of context. If the conference organisers are making requests like this to keep speakers in line because they may not deliver on time, well, that's another issue altogether around who they're choosing to invite to the event in the first place.

Lastly, if something must be provided in advance, crappy web interfaces which struggle with big presos ain't fun. I had to strip out content of a recent one just to get down beneath the size limit! I've had decks with hundreds of MBs (thank you, embedded video) and making assumptions about the size of them or enforcing maximums is just bad news.

3. Asking for slides after the talk

I get asked this a lot and I pretty much always decline. That may sound weird so let me explain:

As I just pointed out in the previous point, slides and anything else that appears on the screen is a very small part of the overall talk. They don't convey the tone of the presenter, their mannerisms or movements and looking at them in isolation without being seen as part of the entire talk is like reading song lyrics without hearing the melody.

I've had cases in the past where people have criticised individual slides after viewing them offline and having never attended the talk. This is from talks that have been very highly rated - sometimes the top-rated talk at major international conferences - because in isolation, the slide is misleading, inappropriate or just plain sucks.

Now having said all that, I'm happy for talks to be recorded and distributed far and wide. That's awesome because it allows people to experience the talk as it was intended to be delivered. It's never quite the same as being there (when I watch my talks later, you can never hear the same levels of laughter and audience engagement), but at least it ensures that everything is seen in context and that makes all the difference.

These first 3 points in this post have actually led me to ditch slides altogether in a number of upcoming talks. I won't say which ones, but they'll be 100% demos and no more slide dramas :)

4. Screens that are only 4:3 (and other AV deficiencies)

There's nothing worse than a beautifully crafted presentation at "normal" resolution (which is 16:9 in 90%+ of cases) then being forced into a 4:3 window therefore adorning it with horizontal black bars top and bottom while losing a big whack of the screen real estate. It's a terrible experience for all and I've often felt that sinking feeling of turning up at a venue and seeing a near-square screen.

In fairness, conference organisers don't always have a lot of choice if that's what's running at the venue already. However, they can definitely flag this with speakers early and as much as I don't want to have to go and redesign my slides, I'd rather that than arrive and have my preso look kinda crap.

Same again if there's not going to be audio out from the machine. That's not very common but I have seen it and it can cause me to either rejig the preso or not show other things on the screen I might have otherwise done. A 16:9 screen and full audio capabilities are the minimum bar and anything beneath that needs to be called out very clearly.

5. Badly prepared rooms

There's a bunch of things that can go wrong here and one is the mic setup. Anything short of a lapel mic is going to make life hard, for example a fixed lectern mic. One of the most valuable speaker training events I attended talked a lot about "stage presence"; how you move, where you position yourself, when you return to the same spot - all of these things matter and accomplished speakers plan this. That goes out the window as soon as you're tied to a single spot courtesy of a fixed mic. (Incidentally, I referred to the resultant lack of movement as "The statue" in my blog post last year about speaker anti-patterns.)

Handheld mics aren't that much better, in part because they restrict the movement of your hands and gestures but also because it makes demos when you return to the machine painful. As a speaker, you want to feel both free and natural (yeah, I know, sounds very hippy-like but it's important) and that means not thinking at all about the mic setup.

Further to that, there has to be an AV person handy to get things like mic levels right and balance audio levels from the machine if you've got sound coming out of there. I've been at an event in the past where a third of my screen was missing in the minutes before my talk was due to begin and I'm literally running around the conference centre trying to find someone to help fix it. That sucked and it was one of several factors that will keep me away from that event again in the future.

6. Conference laptops are (usually) painful

I prepare all my demos meticulously. I have very specific software I need access to. I fire up certain apps I'm going to need before I start. I have backups of my demos. I have other things on my machine I may need access to, particularly come question time. My machine is essential to the success of my talk.

Conference laptops are sometimes provided as an option which is fine, but other times strongly recommended or even mandated. For any speaker doing anything more than a simple set of slides, this makes us very nervous as it's an unknown quantity.

Look - I get it - having a conference machine gives the organisers predictability in terms of all the AV working in a pre-tested fashion and it also means a faster rotation of speakers. There are events where that makes sense; the WIRED security event I did in London in October was a case where it worked well but that was a 20-minute talk where the one demo I had I turned into a video just beforehand.

Particularly for most normal tech conferences though where you've got multiple simultaneous tracks and both speakers and attendees rotating between rooms and sufficient time to setup, give the speaker the choice. Be really cautious about mandating a conference machine because even if they don't outright say it, that's a move that's going to make a bunch of people feel very nervous.

7. Not covering T&E is a massive no-no

When a commercial conference running to make a profit asks me to come and talk and pay my own way there, many words go through my mind. None of them include "sure, that sounds lovely"! I've had this happen multiple times and the invite is usually accompanied by a rationale along the lines of "but think of what it will do for your profile". Let me explain what this actually means:

Every conference talk is a big commitment on behalf of those who take speaking seriously. There are many, many hours of preparation and rehearsals and that's before you even jump on a plane. Then - especially for people like me in a faraway land - you fly for ages and turn up jet lagged. You need time to recover, then of course to speak then get yourself back home again. The bottom line is that it's a one week commitment at the very least when you consider everything and that's a week where speakers can't otherwise earn a living or spend time with their family or do any number of other things.

I had one a little while back where they wanted me to go to Hong Kong and pay my own way. Along with the usual profile-building answer, their rationale was "Bono spoke at the event once". That's great, good for Bono but no, I won't be paying to attend your event so that you can run a commercial show. IMHO, this shows a total lack of respect for the speaker and it implies that the conference is doing them a favour, not the other way around. I'll come back to this at the end, but it has to be a mutually beneficial affair and not covering the bare necessities such as travel and expenses is far from that.

8. Bad hotels are bad

I've ended up in all manner of accommodation over the years and it's ranged from very seedy to very out of this world. Most conference speakers (myself included) have no expectation of the latter, but we certainly don't expect the former. As a speaker, you want to feel like the organisers really value your contribution and the way you're looked after reflects that.

When you end up in a seedy hotel as I did on my last trip, it can really make the event unpleasant. In that case I had no desk to work from, no room to move around should I wish to rehearse and not even an iron to make myself presentable. Other poor hotels have had issues with noise (awesome when you're jet-lagged), no option of room service (same again re jet-lagged when you just want to stay in) or other shortcuts reflective of going budget at the expense of speaker comfort.

Most of the hotels I stay in are fine. Not "great", but "fine" in that they're often small (especially in Europe) and usually pretty basic but they're clean, convenient and offer the basic necessities. Bonus points if it has a bath and I can lie back and soak with a cold beer after a long flight!

9. Not allowing me to arrange my own flights

Many conference events that do cover T&E offer to do travel bookings on behalf of the speaker. I'm all for that with hotels as it's a no-brainer (the issues in the previous point aside), but I've had too many dramas with events booking flights to ever do that again.

The main issue is that once there's you, the event organisers and a travel agent involved, it can be really hard to communicate about even simple things. When it comes to something like changing flights or doing anything "non-standard" with your travel, it can become a nightmare. I've had cases in the past where I've needed to go from one event to another and shuffle my travel which has just turned into an absolute nightmare.

These days, I always book my own travel and invoice the event. I have no problem whatsoever with the event organiser saying "we'll give you $X for flights" and then I'll do the rest. It lets me choose my airline, the ports and times I travel, the class I travel in and most importantly, it gives me full control of any changes I may need to make. Reasonable conferences will support speakers like this.

One caveat on this though: if a speaker is going to be fronting their own cash to travel - particularly when it's internationally - they want to get reimbursed pretty smartly. I've had to chase money way too hard in the past and this should just never happen, for obvious reasons.

10. Not treating conferences as a mutually-beneficial relationship

Speakers get something out of conferences too and whether you call it profile or networking or exposure or whatever, we do it because it offers an upside in our lives. For me, it's all those things and frankly, the excitement of speaking to a large audience too. When I go to a good event that's well-organised and it ticks all those boxes, it leaves a lasting impression on me.

A perfect example of this is the NDC conferences I do in London, Oslo and Sydney each year. Because those guys just get it - I mean everything I've written above - I keep coming back and doing all their events. I've done good talks that rated highly so they want me back and they sell tickets based on my being there (along with the presence of many other top speakers, of course) and I want to keep coming back because they always look after their speakers so well. It's a symbiotic relationship.

I presently have 26 events I've been asked to in 2017 that I've said "no" to and only above half a dozen I've agreed to do. Some of the rejected ones I've done before and had bad experiences with along the lines of what I've written above, others I simply don't know from a bar of soap. But what I will say is that speakers talk; they'll tell others which events really looked after them and which ones were poorly run. For conference organisers, they've got to get this stuff right if they want good speakers standing up there helping to make their show a success.

Because conferences are meant to be mutually beneficial

Those are the big issues that came to my mind after hearing everyone's feedback following that initial off-handed tweet. But I'm also conscious it's very biased on behalf of the speaker and I'm sure there are conference organisers out there that have their own list of ways speakers upset them. Unreliable, unprepared, unprofessional, drunk, etc - there must be a list! If someone writes it then I'll happily share it because I guarantee you, many speakers are far from perfect too.

But if you're like me and you get around to a lot of events, tell me what I've missed. In all likelihood, I'll be pointing conference organisers to this post (and the comments) in the future and I'm hoping this helps them run better events for everyone involved.

Sponsored by: Raygun — Improve UX performance with Raygun Pulse — Installs in minutes — Try free for 30 days!

Last one of the year! And yes, it's summer, it's hot and I'm doing it by the pool. However, as I say in the intro, it's only a fortnight until I'll be back in London which is about as far away as you get in every sense. On a more serious note and harking back to my post on how much effort goes into an international speaking trip, this is well and truly the calm before the storm and things are about to get very, very hard for the better part of a month. But for now, I'm doing my best to enjoy time with family as I hope most of you are at this time of year. Here's the last weekly update for 2016:

iTunes podcast | Google Play Music podcast | RSS podcast

References

1. Lot of data going into HIBP (it takes a long time when you try to handle this ethically...)
2. There's a bunch of ways that conferences can upset speakers (some of these in particular genuinely make it hard to justify doing certain events)
3. Raygun is sponsoring my blog this week - and they rock (no, this is not "say nice things about the sponsor because they're giving me money", I written about them many times before because it's just a good service)
4. I'm off to Europe again! (there's a list of everything I have coming up in 2017 that's already confirmed)

Sponsored by: Raygun — Improve UX performance with Raygun Pulse — Installs in minutes — Try free for 30 days!

I never used to do these "year in review" style things, but 2015 was a really foundational year for me in many ways so I wrote a 2015 retrospective. Thinking about it over the last few weeks as we approached the end of 2016, a bunch of stuff really stuck out in my mind and I think it's healthy to look back at what you've done and take a moment to reflect. Here are the things that were highlights for me:

I launched a new blog

One of the best things I did in 2016 was to re-launch my blog on a brand-new platform with a new theme and wrap Cloudflare around it all. 8 months on, I'm still enormously happy with Ghost Pro in conjunction with Cloudflare caching everything upstream; it just works beautifully. The old Blogger site is now a distant memory, in fact writing this post just reminded me that I could now go and permanently delete it which I've just done so farewell to Blogger!

I got rid of traditional ads

This was another one of the best things I did during the year - I got rid of traditional ads in favour of sponsorship instead. No more trackers, no more other-people's-script-running-on-my-site, no more shitty experiences around flashy ads. The sponsorship messages are just text from 1:1 relationships I have with orgs I respect and they pay significantly more than ads ever did - many times more. So everyone should be happy, right?

Except for ad blockers that strip the sponsor out. I'm still enormously frustrated about this not because of any loss of exposure, but because it's just wrong. Even more wrong are the ignorant comments about "well I as a viewer should be in total control of your content and choose what loads regardless of the consequences". That only works because those people are (fortunately) in the minority. I'm digressing, but it's still a major issue and per the title in the link above, ad blockers are part of the problem.

The Dropbox post went massive

In August, I wrote a post with the simple title of The Dropbox hack is real. It was a simple post where I verified that my wife's 1Password generated very unique, very strong password was stored as a bcrypt hash in the alleged Dropbox breach. The post went massive and was the biggest of the year by an almost five-fold factor:

Not only that, it's almost the biggest post I've ever written accounting for 3.91% of all page views ever (my Shellshock Bash bug post is up the top with 4% of all views).

I had a surprisingly impactful blog post on Ubiquiti

The most surprising post of the year was the one on the Ubiquiti networking gear. This one was quite popular not just in terms of numbers of people viewing it (partly due to hitting the top of Hacker News weeks after writing it), but people then following through and buying the gear. I've had dozens of messages from people that have parted with quite a bit of money to upgrade their networks and I never expected that to happen:

Finally got my wifi fixed. Thanks to @troyhunt and his post on @ubnt hardware. The financial damage was worth it :) https://t.co/62CD53ip7H

— Ingo Bente (@ingobente) December 24, 2016

@troyhunt @ubnt U inspired my to buy and write about my setup. It it writen in Icelandic like all good things :Dhttps://t.co/a9gXzAoJk5 pic.twitter.com/P4yXWFr43s

— Jon Olafsson ⚡️ (@jonolafs) December 12, 2016

Boy is my wife going to be mad at @troyhunt when she gets home... 😂 pic.twitter.com/Xe465BQatn

— Scott Helme (@Scott_Helme) November 28, 2016

And no, I'm not on the payroll and in fact, I'd never even spoken to the folks there before deciding it was gear I should buy then loved it enough to write about it. That's the sort of post I love writing!

I started doing weekly update videos (and a podcast)

This was an idea I'd toyed with a bit as a means of trying to add some more candour and emotion to a lot of what I was writing each week. My mate John Sonmez gave me a bit of a push and that was it - I was off - and I published the 15th edition just a couple of days ago. Since I began, there's been thousands of people a week either watching the video or listening to the podcast which I'm enormously happy with.

These are easy for me to do (other than the difficulty of uploading 1080p video on a connection that barely gets 2Mbps up on a good day...) and I genuinely enjoy them. The feedback I'm getting is that people like the ability to consume this information in the background which was one of the reasons many people asked for a podcast version. I honestly think I can do a number of things better, but it's early days and I'm pretty happy with how it's going so far.

Have I been pwned (HIBP) grew enormously in size

It grew in every measurable way:

1. The total number of breaches in the system went from 67 to 178
2. The data went from 256 million breached records to over 2 billion
3. The number of verified subscribers to the service tripled from 316k to 942k
4. The Alexa rankings took it from 60,000th most popular site on the web to about 36,000th today (it peaked at about 20,000th)
5. The Twitter followers grew from about 7.5k to almost 21k today

But the really big numbers were around the traffic...

The HIBP traffic got big - real big!

One of the biggest changes this year in terms of traffic was the emergence of large volumes of requests that could be equally classified as abusing the API, malicious or in some cases, outright attempted DDoS. It meant traffic patterns like this:

Still seeing stupid traffic patterns on @haveibeenpwned, this just seems to be the new normal (basically bots getting blocked at Cloudflare) pic.twitter.com/DClWcrRTUw

— Troy Hunt (@troyhunt) November 12, 2016

That gives you a good idea of the distribution of the traffic across a day (i.e. it can be concentrated within certain hours), but that was far from the busiest day too:

This is the most traffic I've seen directed at @haveibeenpwned in one day - over 121M requests, almost all from @Cloudflare cache pic.twitter.com/no3fSl9mrz

— Troy Hunt (@troyhunt) November 4, 2016

Wrapping Cloudflare around HIBP was one of the smartest thing I've ever done with the service. A combination of that and the way I'm using Azure Functions to control firewall rules has made a hell of a difference. I'm writing up a detailed post on how I dealt with malicious traffic and how everyone else can plan for it, suffice to say that without Cloudflare in front of it I just couldn't have done it without making some major concessions.

But big traffic wasn't always malicious either; there was the case of The Martin Lewis money Show causing a massive spike:

So @haveibeenpwned just went from 200 users on the site to this in less than 60 seconds due to a mention on the Martin Lewis Money Show: pic.twitter.com/IsPODLgvpM

— Troy Hunt (@troyhunt) November 28, 2016

That was a great learning experience actually and again, Cloudflare made a big impact on the traffic volumes I could handle:

Good illustration of the @Cloudflare value: @haveibeenpwned got knocked offline for a couple of mins, but only 24% of traffic hit it: pic.twitter.com/3V7j7lrkRV

— Troy Hunt (@troyhunt) November 28, 2016

What all of this means is that whilst visitor numbers (both good and bad ones) have gone through the roof, my underlying infrastructure costs haven't changed which makes me enormously happy!

I spent a third of the year travelling

This is the one that really dominated in terms of total effort and the stats tell the story better than words can:

When I saw those numbers, I thought "wow, that's a lot" then I checked back on 2015 and it had been 240 hours in the air across 48 flights, but it just felt harder this year. This figure from TripIt explains why:

Almost one third of the year was spent away from home. Now in fairness, that wasn't all hard work; I had a week in Oslo with my wife (although we were both at NDC speaking) and a week snowboarding with the family, but it was still a hell of a year. I wrote about how much effort goes into an international trip after my last one and frankly, that's probably the hardest one I've done. The next one in a couple of weeks will be a similar length, albeit only 3 countries with one uninterrupted week in each which makes a lot more sense.

I went to a lot of conferences

Speaking wise, I got around a bit this year:

I keep track of all these on the events page for the year and I also publicly share all my evals too. I did a lot of events, but the highlight was doing the opening keynote at NDC in Oslo in front of thousands of people:

It's a special event for me because NDC Oslo was my first international speaking appearance back in 2014, a talk that topped the ratings and has had me coming back to every NDC event ever since.

Another 9 Pluralsight courses down

It was a big year Pluralsight wise, not least of which because I finally finished off the epic Ethical Hacking series. That was a mammoth task, not just in terms of writing 8 courses over 2015 and 2016, but writing them to the CEH syllabus so people can then go out and get their Ethical Hacking cert. The link above explains more and I'm enormously proud of what we've created there.

I also found myself doing a bunch of "Play by Play" courses, that is courses where myself and someone else are video'd working through a technology. I did one in London, then Chicago and finally a couple in Sydney, not all on security either. I've got another couple to do in London in a couple of weeks' time too, neither of them about security and you may well see me diversifying a little bit more there in the coming year too.

I did a heap of workshops

I'm not actually sure how many workshops I did in 2016. Probably 20 events of 2 days each? I'm not sure but what I do know is that I'm enormously happy with how they've been going. I'm now doing a number of repeat events for the same organisations as they expand their security training or even dive deeper with the same participants who've been to previous events.

I also love that many of the organisations I visit have already invested a lot in Pluralsight. Very often, those who attend the workshops have seen many of the courses I've written but they want to augment online learning with a classroom environment. It surprised me at first, but when I see how these organisations run their training and how they draw from the strengths that both on-demand remote learning and in-person events offer, it makes a lot more sense.

I'll be doing a heap more events in 2017. I got overbooked by 4 events on the trip I'm about to do and will shortly be sharing plans to pick up that overflow (and a lot more) on a subsequent trip so stay tuned for that.

The Microsoft Regional Director thing happened

A very unexpected outcome of 2016 was becoming a Microsoft Regional Director. Because it wasn't already confusing enough that I don't actually work for Microsoft, I obtained a title that leaves people even more confounded by the whole thing!

The RD title is something I'm enormously proud to have received and it's something that along with being a nice recognition, opens doors. Particularly in corporate scenarios, it carries a weight with it that goes a long way in terms of credibility and from a personal career growth perspective, it's a great thing to be able to say I've achieved.

Social media continued to be enormously important to me

In very unexpected circumstances, I posted my most ever favourited and RT'd tweet:

This car is white pic.twitter.com/1DNF8rg2M0

— Troy Hunt (@troyhunt) September 12, 2016

All those years with all that work actually building software and writing about constructive things yet it's an off-chance photo I snapped which gets the popular vote!

I've had many enormously positive experiences by way of social media and arguably Twitter in particular has had a huge impact on my career. It's not just in terms of reach on the internet either, it's been a great way of connecting with people that's led to in-person meetups which I've been doing more and more when I travel.

Particularly in my independent life these days, the ability to reach people is enormously important and Twitter in particular has been invaluable for that. People often ask how I promote what I do, for example how I book commercial workshops, and the answer is simple - I tweet about it. That is all. Seriously, that has been more valuable to me than just about anything, but it only makes sense off the back of a good reputation...

My profile grew (and the trolls continued to circle)

Profile is always a bit of a funny one because it's a little bit frog-in-boiling-water; it happens gradually so that you don't notice it yourself. Every now and then you get a wake-up call (such as people wanting to take selfies at conferences) and it's honestly a very strange feeling to be "famous", even if it is just within certain circles.

I put a huge amount of thought into how I curate my profile and each year it becomes more and more important. In 2016 I thought a lot about the balance of how I can both use the profile to my advantage (i.e. by offering the blog sponsorship) and keep it down to earth and, as we'd say here, fair dinkum. I don't want to erode what it is that's helped me build that profile in the first place, namely being approachable, ethical and remaining very practical and hands on (at least I think they're the things that led to where I am today). That's not always easy though and a perfect example is that I got to the realisation this year that I simply can't reply to all emails or tweets or other communication channels people reach out by. I added a contact page to my new blog where I literally had to say "here's a bunch of things I either may not or will not reply to", and that includes some very genuine enquiries from people.

It hasn't all been roses though and a few months ago, I wrote about online abuse. I still find it hard to fathom that as an adult, you can be subject to what I can only describe as playground taunts. And really, that's what a lot of this is - things I teach my kids not to do - yet here we are with everything from name calling to slanderous comments to actual threats. Even since I wrote that post there have been incidents; never in person, mind you, because the weak mettle of those involved keeps them firing barbs from a distance, even when the opportunity has been there to look me in the eye. I think what's bugged me more than any hurt feelings is just the frustration that people like this walk among us and seem to be oblivious of their own behaviour, yet here we are.

On balance though, I wouldn't have it any other way. Building profile had brought so many wonderful experiences and new connections with people and places I just never dreamed of. I love that there are no limits to it - you can grow as big as your own hard work permits - and that's a key factor that keeps driving me forward.

Looking forward to 2017

The trick for me this year more than ever is juggling priorities. There's Pluralsight courses, HIBP, workshops and, of course, speaking events. As of today - the first day of the year - there's already 26 events I've either declined or put on ice:

I won't show you precisely what they are as I don't know how many of the organisers would like to share that information, but obviously that's a fair number. It's hard because I genuinely want to get to many of these but I also like actually seeing my family!

Speaking of which, just today I booked my family to come to Europe for a couple of weeks in June around the time I'll be there anyway for the NDC conference in Norway. I'm going to make a bigger trip out of it than usual and both spend some time with them (mostly in Oslo, Amsterdam and London) as well as try and do more events. I'm already starting to book things in for that time period so if you're in Europe and are interested in your company having me over for a private workshop, speak up now!

Lastly, for 2016 and for all the years to come, I'm enormously appreciative of everyone who reads what I write, watches my courses and listens to what I have to say. That's what makes it possible for me to do what I do and have the wonderful opportunities I touched on above. Thanks everyone!

Sponsored by: Raygun — Full stack error and crash reporting for web and mobile — Installs in minutes — Try free for 30 days!

I don't mind ads on websites as a concept, that is I don't mind the idea of a message appearing somewhere that helps the producer of said content earn a crust. However, there are other things about ads that I do mind enormously and most of them are due to the ad networks themselves. I don't like the overhead of a whole other website being embedded into an iframe. I don't like the total irrelevancy of much of the ad content. It could be tailored to my browsing habits, but then I'm not overly fond of the tracking. Oh - and I definitely don't like being served either malware or really obtrusive behaviours such as ads viewed on iOS redirecting me to the app store in an attempt to have me download Clash of Clans. None of these things are fun and they're all directly attributable to the way networks run.

Back in September, I'd had enough and I introduced sponsorship of my blog. Now I don't care if you call this an ad or not, but what's fundamentally different is that it amounts to nothing more than a single line of text at the top of the site that I serve myself. For example, here's what viewers today are seeing:

Raygun is a perfect example of the sort of sponsor I want on board too because I've written about them many times before and I'm honestly, independently endorsed in what they do. In fact, every sponsor I've had on board to date has done things that are entirely relevant to my audience which is a really nice way of doing targeting without all the invasive stuff that goes along with it. Partly that's because I curate them - I've rejected multiple potential sponsors because our views simply weren't aligned - yet even still, I've never had an un-sponsored week since I launched. Actually, all sponsor slots up until the beginning of April are now fully booked out, and that got me thinking.

When I launched the sponsorship model, I assumed I'd have plenty of weeks where there wasn't one. In my naivety at the time, this seemed like a reasonable assumption so I had a fallback position that would revert to the old ad model if the slot wasn't filled. It would also mean my sponsor bar would look like this:

But thinking about it lately and considering both how well the sponsorship was going and how much I dislike ad networks, I've decided to permanently kill all ad network code. HTML - gone. CSS - gone. JavaScript - gone. If I have an un-sponsored week then that last image above is all that will happen and that's absolutely fine. The sponsorship model has worked enormously well in every way and that includes financially, so I see no sensible reason why there should be any remaining ad network remnants whatsoever.

We need a better commercialisation model for content creators that doesn't suck like ad networks do. There has to be a happy middle ground somewhere between all the nasty things that (most) ad networks bring with them today and the other end of the extreme which is outright blocking of ads. And honestly, they've got issues too - I recently wrote about how ad blockers are part of the problem and whilst some people vehemently disagreed, I stand by that assessment.

What I'm trying to do with sponsors is what I strongly believe is the most responsible middle ground that keeps the greatest number of people happy, myself included. Thank you to everyone that's been supportive of this model and of course a big thanks to the sponsors as well. As of now, all ad network code has been permanently removed from this site!

Sponsored by: Raygun — Full stack error and crash reporting for web and mobile — Installs in minutes — Try free for 30 days!

It's a new year! Which means looking back at the old year and while I'm there, also looking back at how much we didn't know we didn't know. This week I also permanently nuked all remaining remnants of the ad network given the success of the sponsorship model and that has made me very happy. What I didn't mention in the weekly update is that I've had over 70k visitors to this blog over the last 24 hours largely on the basis of that post. It got a lot of traction on Hacker News which obviously helps (but wow, some of those comments...), but it's interesting how much of a hot topic ads always tend to me.

iTunes podcast | Google Play Music podcast | RSS podcast

References

1. I looked back at 2016 in a retrospective (seriously, do this in your own lives as well, it's good for you!)
2. There are known knowns and there are known unknown, then there are unknown unknowns... (Rumsfeld sounded kinda funny all those years ago, but there's some truth in that saying as it relates to infosec)
3. Ads are gone - I mean really gone (all last remaining code artefacts have now been permanently banished!)
4. Come see me in Copenhagen! (there's still some seats left for my workshop at the end of the month)
5. Raygun have sponsored me for the third week in a row (I genuinely use what these guys do a lot so check out their good work)