back
134 comments
Textbook on how not to handle the issue. Deleted Twitter, sanitized Facebook and more than 2 days to admit the issue.

There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available.

https://twitter.com/ConleyU/status/1151862278909825024

https://twitter.com/MRasconCPA/status/1151894366291734533

https://twitter.com/hockeygirlPDX/status/1151945932935585792

Ouch. This is the sort of stuff that can kill a company.

Does Quickbooks with the cloud option offer local backups?

It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data.

Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.

QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups.

There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.

Just a matter of time for ransomware to replace data via APIs.

As the resident tech guy in a small shop with Quickbooks, yes, you can do local backups. We're not using iNSYNQ for cloud hosting, though.
This is a question I've thought alot about, so many some Sys Admins can give a good idea about how to approach it:

How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.

A generally safe backup process looks like this:

Production has no access to backup.

Backup has read only access to production.

Backup writes are append and not overwrites.

Deletes/archival are governed by a retention process.

Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.
ZFS Snapshots are really handy - it takes root access to remove them and you can't delete them with the normal UNIX delete. IF you're sharing with Samba or NFS, there's no way for any remote user to be able to delete.

You just go back to the last good version.

Make the backup servers pull?
The easiest solution to this is out-of-band backups happening at the storage layer. The details depend heavily upon your platform and storage hardware, but most SAN and virtualization solutions today allow one to backup hosted systems without being accessible by those systems (and no agents installed into the guest OS).
You pull from the production servers rather than push to the backup servers. That way even if the production servers are compromised your backups are still safe.
Unless you have persistent mount to network FSs like NFS or SMB how do you think the ransomware would spread? You sure don't need network mounts for backups.

Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).

Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.

I'm not a sysadmin, and I'm pretty sure that <big company> does it different, but here is how I do it at home:

- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.

- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.

- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.

This setup let me sleep pretty well.

Drawbridge, you configure a firewall to allow access only at certain times. Also you can configure a NAS so that deleted items are preserved without a admin removing them.
I use FreeNAS (FreeBSD NAS with ZFS). The ZFS snapshotting feature effectively gives me protections from this type of issue.

My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.

This effectively creates an append-only backup NAS thanks to the periodic snapshots.

In the past I had the backup server ssh via public key into the production server and pull the data via file system snapshots. It’d be a matter of restoring the most recent snapshot prior to the attack.
You can write backups to an NFS share, and create regular snapshots on the file system server that are not writable from the outside (you have to log into the file system server, which hopefully is extra secured / limited access, and has a different OS than what you use elsewhere). It's not perfect, but at least a single vulnerable client can't encrypt or delete your backups (in the snapshots) that way.

As usually with security, the principles of least privs and segregating as much as possible are important.

Append only?
Windows strategy: one way dfs replication to onsite backup server hooked up to an offsite cloud backup provider.
Make S3 bucket on a different account, grant it write privileges from your main one. Enable versioning and setup lifecycle to purge old versions in 90 cays. That’s it - no matter what your main account does, you will bave 3 months to undo it
Backup could (should?) be on tape, which once written is physically removed from the system and put in cold storage for some period of time (years?). That's how we did it at my first job, at least.

Of course IANAL so idk how this jives with various EU laws.

Most attacks are done via local file access. Ship things over an API.

EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.

Also how do you prove integrity of your backups if you don’t have certainty of initial perhaps temporarily dormant breach? I mean your backups to tape could still have bad data.
you can use s3/gcloud files with "create"/"read" only rights.
iNSYNQ is a third-party service platform operator who specializes in hosting instances of QuickBooks in the cloud.

This does not affect non-iNSYNQ QuickBooks instances, such as those operated by Intuit (the creator of QuickBooks).

I had no idea this was how QuickBooks even worked in the cloud. Could Intuit be in any way liable (like is this a sort of franchise type of thing)? Or is it closer to Word Press where different companies can install "QuickBooks Cloud" and then offer it?
Looked for but didn't see this when I made my shockingly similar post.
It's not immediately obvious, but iNSYNQ provides hosting for QuickBooks Desktop as a virtual desktop service (think VNC or RDP). This is distinct from Intuit's own QuickBooks Online cloud service, which I believe is unaffected by this breach.

I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.

I can confirm QBO is up (I am in Canada). Holy Batman, the chaos if that data would be gone... I did a backup, quickly. I need to ask my accountant whether they back this data up regularly, if not then I need to. I am a very small company but my invoices are international and while I have my invoices at hand, if I would need to reconstruct the exchange rates for an audit a few years back... shudders
Was this service really less expensive than just using actual QB on an RDP available cloud server from Azure or any number of other services?

Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.

In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.

> really surprised Intuit doesn't have this as a cloud service at this point.

They do have a cloud offering, QuickBooks Online:

https://quickbooks.intuit.com/online/

But it does not have all the same features as the Desktop version, giving rise to a number of third party offerings, like Right Networks' "QuickBooks Desktop Cloud": https://www.rightnetworks.com/cloud-solutions/accounting-sol...

"After the third day of outages, customers were saying Bye Bye Bye."
One company was already in the process of moving after iNSYNQ got hit earlier this year by ransomware. This isnt their first attack.
It's not impacting all of Insynq's services. I work with an Insynq customer. Their Insynq services are still running, by some generous stroke of fate. The only outage we noticed was in the middle of the day on the 16th. For about an hour, users weren't able to access the service. I called support, and was diverted to a recorded message saying that they were doing normal maintenance, would be finished shortly, and were aware of and sorry for the disruption.

It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.

Wonder if they obtained any people's financial data or social security numbers. Probably mostly self employed people and small businesses. Pretty scary how people use their SSN for everything. Seems so insecure to have a number you just openly pass around... Need to get a ID or license, credit card, bank account, your doctor, dentist asks for them, your phone company, cable company, of course when getting paid, and even police officers ask for them sometimes and write down in a notepad if your name happens to match someone else's name who has a warrant. Many other uses probably too I didn't think of off the top of my head.

I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.

Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.

I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.

> I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning...

This is for spousal rights - i.e. if your spouse is allowed to request access your data.

> I'm surprised banks haven't figured out a way to offer deposit only option

They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.

For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.

The cloud is just someone else's computer.
The article seems light on details about how QuickBooks is involved. Does iNSYNQ host QB Cloud data for them, or does iNSYNQ white-label QB software?
What are the odds their vulnerability was due to IT staff choosing to not patch their Windows servers, as doomed Baltimore in May? https://krebsonsecurity.com/2019/06/report-no-eternal-blue-e...
Is there a way of freezing a compromised machine, such that the contents of RAM are kept around for forensics, but the compromised system can't do any more damage to itself? SysRq? Intel ME?
Slowly but surely ransomware is becoming an absolute threat to all data. It seems more like not if but when it will hit.

It seems to me that it's time the OS providers start providing a very easy way to restore the state of data. We all know that backups are the answer but as long as people, have to think about it, there will always be some that don't do them. And now that you can get a 1TB HD for less that $100 then it's a no-brainer.

Virus protection is now automatic with Windows when will backups become automatic on all OSs?

Why would someone use a third party to host Quickbooks? Won’t Intuit host it?
If anyone wants a recommendation for an alternative, I can happily suggest Xero.

What a mess, though. Worst part of a business to be crippled is its core - financials. A part often overlooked by techies. If you can’t invoice, you can’t pay the wages.

For an accounting software this is disaster.
My Quickbook Self-Employed (hosted) seems unaffected (at least so far).
I lost all interest in the topic when I read the hosting companies name, can we just talk about THAT? Like, really? Who picks that name for their company, certainly the band nsync has to be older than the company.