back
137 comments
I'm now expecting we'll see a couple things in the next few years:

1. An explosion of residential proxy networks and other stuff to circumvent blocking of cloud IP ranges, for all the various AI scraping tools to use.

2. A corresponding explosion of countermeasures to the above. Instead of blocking suspicious IPs, maybe they get a 3GB file on their request to /scrape-target.html

Perhaps an explosion of usage. There's already a few very large residential proxy networks.
I think that may be against the ToS of most residential ISPs.
200GB is nothing since 2018 when AT&T mass introduced their 1-gig symmetric fiber. Any single common gigabit link can run 200GB in 15 minutes.

On any gig link, over the course of 6 hours you can transmit a little more than 4TB one way.. which is 40x more.

Too bad AWS didn’t get that memo, 200GB would cost $18 there, and somehow the company in the original post is paying $500 for that bandwidth with whoever their proxy host is.
Haha unfortunately we use residential proxies under the hood to simulate real users (as you'd expect from AI agents), where bandwidth is significantly more expensive!
The screenshot is from webshare
The discussion linked in the post is from 2022, and the corresponding issue has already been fixed:

https://issues.chromium.org/issues/40220332

I wonder if there is a more recent bug related to this?

I think that Chrome is still doing a lot of downloads. They're just no longer showing them to the user.
And uploads.
Gosh I regularly burn through that much just updating games in steam :D. Not proxy bandwidth of course but isn't it funny that the the line between regular usage and $$$ can be what is using the bandwidth. Or rather, isn't it funny that regular consumers expect to be able to use multiple terabytes of data for < $100/mo but the same can still be thousands in other enterprise domains
>Gosh I regularly burn through that much just updating games in steam

Same, and I get angry letters from comcast about abusing my 'unlimited' bandwidth.

200GB for $500? What cloud is this?
I don't think it's a cloud. It's more likely a residential proxy network, which are typically created by installing malware on users' machines.

The operators of these proxy networks want to avoid detection by both the users whose bandwidth they're stealing, and by the companies whose data is being scraped. So they want to make the bandwidth very expensive. And that expensive bandwidth in turn means that their only clients are dodgy as well. Either people looking to scrape data without consent and monetize it, or outright criminals.

Residential proxy service

https://smartproxy.com/proxies/residential-proxies/pricing

(may not be this service, but this is an example, and the price is consistent with their larger commitments)

Absolutely wild. A normal price for bandwidth before volume discounts is 1c/GB, or 10 bucks per TB
api.skyvern.com is a CNAME to an EC2 ALB, but even using a NAT Gateway ($$$) I can't make more than $1/GB add up.
Looks like webshare from the screenshot
I downloaded world of Warcraft the other day, 100GB, took less than 3 hours and you can be sure it didn’t cost blizzard $0.05.
I would have liked to see a bit more of 5 Whys here. It seems like a consistent lesson that startups have to learn over and over is how to manage external dependencies, and particularly the dangers of having Google as a dependency. This is new Chrom(e|ium) behavior, and it has a real cost, both for this company and for users, which may or may not be worth the ROI, but this is what happens when you have a large scale external dependency: stuff moves without your knowledge, consent, or control.

Instead of Always. Be. Closing. it should be Always. Be. Mitigating. Dependencies. for startups.

This is a great callout.

We had an internal discussion about how to manage dependencies effectively, and we made the decision accept the risk that comes with blindly relying on Chrome for now, instead of investing heavily in mitigating that risk today.

The main motivator was for us to continue moving fast, and accept that we have a few hard dependencies in our business.

The goal is to find product market fit, then allocate time to de-risk some of these hard dependencies. If we fail to find product market fit, this may not matter at all

What infrastructure is this using? Bandwidth seems pretty pricy
No kidding. AWS's notoriously expensive data transfer is only $0.09/GB. Who's charging $2.50/GB? Are they running on a cellular SIM with no data plan?
" 200GB of proxy bandwidth was approximately $500 burned over the course of 6 hours"

The fuck ? So Internet is literally more expensive than buying a drive at amazon, paying for shipping, filling it up putting it on a truck towards a destination anywhere in the world.

Well, one part of the source of the problem is this, where I not even understand all of the words (a bit exaggarating):

> Skyvern is an AI agent that helps companies automate workflows in the browser. We run leverage proxy networks and run headful browser instances in the cloud to facilitate most of our automations.

So you're doing Selenium, just with Cloud, AI and some other buzzwords you found while Googling?

I mean, yeah, bandwidth costs aren't just about bytes, they're about energy, infrastructure, and routing complexity too.
Literally means cloudprotection in Norwegian. Thought for a second we had gotten our own cloudflare.
I was so surprised to read this, especially since we definitely didn't know about it when naming Skyvern.

Out of curiosity, I checked both Cambridge's Norwegian-English dictionary and a few other Norwegian sources, but I couldn't find 'Skyvern' listed anywhere. Makes me wonder if Google translate just hallucinated the meaning.

Please, gigabyte isn't a unit of bandwidth.

Bandwidth is measured in data/time

Tell that to every single ISP and Cell provider.
"We run leverage proxy networks and run headful browser instances"

Um...say what? I'm pretty broadly based in IT, and I have no idea what that means.

Haha, apologies for the language!

We use residential proxy networks when running Skyvern to help simulate real human behaviour (because that's what Skyvern is trying to do).

We run headful browser instances (meaning a real chrome instance running with a real viewport) for the same cause!

Skyvern is a great name, very evocative. Typical arrogant Google, downloading trash to the user without consent.
Honestly given many of these stories, $500 seems to be getting off pretty lightly.

It’s still absurd to me that many (most?) of these hosting/bandwidth providers don’t seems to allow automatic cut offs and such

It definitely could have been much worse. We burned through our monthly allocation in 6 hours HAHA, I'm grateful that our allocation wasn't something like 10TB
Hello,

A (different) proxy company owner here. This sucks! Sorry that you lost out on so much bandwidth.

Feel free to reach out to me at tim@pingproxies.com and I'd be happy to get you set up on our service and credit you with 100GB of free bandwidth to help soften the blow. I'll also be able to get you pricing alittle better than you're currently on if you are interested ;)

Within the next few months we're also releasing a bunch of tools to help stop things like this happening on our residential network such as some intelligent routing logic, spend controls and a few other things.

You may also want to look into Static Residential ISP Proxies - we charge these per IP address rather than bandwidth and they often end up more economical. We work with carriers like Spectrum, Comcast & AT&T directly to get IP addresses on their networks so they look like residential connections but host them in datacenters - this way you get 99.99%+ availability, 1G+ throughput, stable IP addresses and have unlimited bandwidth.

@ everyone else in the thread; if you run a start-up and need proxies then email me - happy to credit you with 50GB free residential bandwidth + give some advice on infra if needed.

Cheers, Tim at Ping

I’m interested to know how your residential connections are sourced.

It says they’re “ethically sourced”, but it seems like malware/botnet like behavior.

Are these residential users aware their traffic is siphoned off for this purpose?

Blocking Google from downloading anything onto your computer without consent is always a good idea.
We were pretty careful about what we were blocking here -- had the exact same concern. Hopefully it doesn't come back to bite us in the future (new blogpost incoming?)
Especially if you're using expensive bandwidth from botnets.
>200GB of proxy bandwidth

Gigabyte is a measure of information.

Bandwidth is information transmitted over time.

you shouldn’t be paying by the terabyte. Colocate and just pay for the maximum throughout. Far better rates
doesn't work when the sites you're scraping block the IPs/range of your server. They're using a proxy botnet that costs a premium