AWS Transfer for SFTP – Fully Managed SFTP Service for Amazon S3 (2024)

There are a handful of services out there that do this, I know because I've needed it on multiple occasions. It's nice that Amazon is providing it in house now, but it just reminds of me the last time I went to re:invent and walked through the vendor area and thought about how many of these companies are four dev cycles away from Amazon producing a baked in competitor.

Hard to make a B To B Amazon tool these days.

I wrote a part of our product, actually invested years in it, that does a certain thing I won't say on Amazon's cloud. One day, Amazon released a competing service that does the same thing.

There was a flaw in our product that had one of our customers pushing to use Amazon's offering. Turns out Amazon's service cost about 4x what it cost us to do it ourselves. It wasn't obvious at the time given Amazon's purposefully obtuse pricing.

Eventually we fixed the bug, brought our customer back in line with the rest, saved some money, and have continued providing this service much cheaper than Amazon. I think it works for us because of a few reasons:

1. The service we offer is challenging enough that most dev teams won't want to do it themselves, they'll outsource it

2. Amazon has little incentive to charge less (see #1), little competition

3. We're small enough that we can still provide that face-to-face level of service and hand holding that's nearly impossible to get from a larger org

Amazon/Microsoft/Google may come into a particular market, but it doesn't automatically imply that they can (or even will) do it better/cheaper/faster.

Dave Winer:

http://scripting.com/stories/2007/04/04/coexistingWithPlatfo...

Sometimes developers choose a niche that's either directly in the path of the vendor, or even worse, on the roadmap of the vendor. In those cases, they don't really deserve our sympathy. It's almost like a game of PR, there's no way you're not going to have a fight on your hands.

Further discussed by Joel Spolsky

https://www.joelonsoftware.com/2009/06/10/platform-vendors/

Thanks. At the end of Spolsky's article he mentions a way to survive:

"A good platform always has opportunities for applications that aren’t just gap-fillers. These are the kind of application that the vendor is unlikely ever to consider a core feature, usually because it’s vertical — it’s not something everyone is going to want. There is exactly zero chance that Apple is ever going to add a feature to the iPhone for dentists. Zero."

That’s quite literally AWS’s playbook. Fill the vendor hall with tech, learn it, see what sticks and then crush it.

This is why I’m bullish on cloud agnostic tech. These practices don’t typically fair well in the enterprise space. This is why companies like MSFT are interesting to me. They partner and rarely kill. Amazon is the complete opposite.

You can't even sell merchandise on Amazon without significant risk they will like your numbers enough to muscle you out:

https://www.amazon.com/AmazonBasics/b?node=10112675011

The mini condenser microphones are actually REALLY good and on par with Yeti and other $100+ mics.

https://smile.amazon.com/AmazonBasics-Desktop-Mini-Condenser...

According to reviews that's a lowly electret microphone capsule ($1) in a big housing. For podcasting you want a directional microphone. In defense of electrets, they sound great, but pick up everything.

The shape of the microphone suggests it's a directional microphone (and if you read the reviews, people think it is directional). That's perhaps not a scam, but certainly deceptive.

> They partner and rarely kill.

How the times have changed.

Who can forget "don't let the haircut fool you, i'm exceedingly wealthy": https://www.youtube.com/watch?v=H27rfr59RiE

I wouldn’t count on that. These hyperscale players don’t have the time to worry about their “friends”.

Example: Do you really think that the warm embrace of VMware Horizon and Citrix on Azure is the beginning of a long and fruitful relationship?

That’s a fair point, but I would look to specific markets instead of tech to see where AWS and Amazon might run into issues.

Specifically within healthcare. You think large Pharma companies will use AWS after the pill pocket acquisition? Do you think once Amazon.com starts listing prosthetics large life sciences companies will run on AWS? What about providers? Do you think health systems will choose AWS as their cloud once Amazon launches their version of KP?

I don’t see Azure or GCP getting into these specific markets.

I would agree that it's unlikely that Google or Microsoft will start making prosthetics -- although they could spin GCP or Azure off into other corporate entities or operators who did someday.

More likely would be that Google, Amazon, Microsoft and some affiliated company would be competing in a space where telemetry from something like a prosthetic was reporting information that had some value.

In any situation, the downside of renting something is that you lose control. It is something that you need to think about and incorporate into your business strategy in some scenarios.

My company, Thorn Technologies, has a direct competitor on the AWS Marketplace called SFTP Gateway. We haven't seen any negative impact on sales just yet, probably because our product is much cheaper, and we think better. But only time will tell!

Yeah, this is so typical of AWS to do this, and I know we're not alone.

An alternative to doing file based SFTP is to just treat SFTP like an API.

A company I work for implemented an SFTP service where every operation simply translates to some SQL DB lookup. And a file download kicks off a larger SQL query and generates the report on the fly, streaming the result straight through to the SFTP client.

Works great! SFTP can be an API just like HTTP. Under the hood the protocol is reasonably contained and doesn't require a filesystem backend at all.

Depends a lot on the usecase of course.

See https://github.com/pkg/sftp

From parent: "Depends a lot on the usecase of course."

The usecase that I see most often of SFTP (and hinted at in the parent's problem description) is generating one-off reports for third parties, or passing data to vendors who are stuck in the 90s, like financial services companies.

It's almost always read only (or read and delete), in which case implementing an API like this is pretty straightforward. Log unsupported commands perhaps and decide if you want to implement them later.

I think this implementation uploads it to memory before going to S3. It usually won't handle 20GB files (unless you have like 20GB of RAM) and in this case, were it a smaller file, it'll just never upload.

https://github.com/moriyoshi/s3-sftp-proxy

>There are so many legacy app flows kicked off via sftp/scp file drops

Yes,.... Legacy apps... because no one would choose SFTP for system that designed in 2017.

Seriously this is great, so many solution rely on SFTP, but so many companies fail at managing the service. Having an SFTP service that just works and is secure (hopefully) will help a ton of compnies.

The only downside is whitelisting but not on the SFTP server-side. Many enterprises restrict egress SFTP (usually for security reasons) so you need to provide IPs and they can’t frequently change because it can take enterprise network admins quite some time to deal with all of the bureaucracy and change control.

That said, I wouldn’t be surprised if modern networking gear can handle CNAMEs but there’s no guarantee that they’re using modern gear or if they are that the questionable outsourced team even knows how to deal with the modern capabilities.

This will certainly help a lot of use cases though.

It's less whether the gear is modern and more about the layer that it operates at.

A network firewall doesn't see the DNS name that an internal system looked up in order to make an outbound connection. It just sees the source/destination IP/port. Processing a rule based on source/destination IP or CIDR and port is very fast, and all happens locally. Trying to make that device handle rules by IP address is pretty tricky. Does it do a reverse lookup on the destination IP? That may not give a result that's even remotely like what the client used, especially for cloud-hosted destinations.

For a lot of applications (probably including this one), a proxy is a good approach, because DNS resolution can be delegated to the proxy, and therefore the proxy can easily apply DNS-based rules as well as IP/CIDR-based rules. However, proxies tend to make people unhappy because they generally require at least some configuration on the client side. Microsoft used to sell a product[1] that made this transparent for Windows clients[2], but obviously that doesn't help for most modern shops where a lot of the systems are Linux, MacOS, etc.

[1] Internet Security and Acceleration Server ("ISA"), later renamed to Threat Management Gateway ("TMG"), now deprecated and approaching EOL.

[2] It hooked into the network stack and rerouted requests based on a proxy routing rule table. Imagine a centrally-managed proxychains, but with the system configured to default to check the proxychains config file for every outbound TCP connection.

I'd be curious how this handles all the posix cases not well suited to object storage.

Renaming a folder than has a million files/folders inside is a single operation in SFTP, but 2 million operations on S3.

Does it handle writing at arbitrary offsets within a file? Does it download the file first then let you start writing?

What about just writing a few bytes at the beginning of a large existing file and then closing your SFTP handle?

How about 2 users accessing same file via SFTP at the same time?

I don't work on the S3 team so I can't answer all these questions but relative to the "2 million operations on S3" question I can point out that S3 now has batch operation support: https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-...

Charged by the hour... pay for an instance.. Tsk tsk..

The thing I love about S3 and cloud services in general is when I pay per request and can scale through the roof.

Whenever a services is meter by number of instances my interest fades, and I look for other solutions..

S3 has this very handsoff feeling to it :)

Sure, it's because an sftp server is always running.

But I immediately extrapolate that this also means it has bandwidth limits and limits on concurrency, etc.

We currently pay $250/month through some small vendor for hipaa compliant sftp hosting (that we transfer a whopping 50kb on a weekly basis). I always felt like it was a rip off, but azure/aws didn't have their own version. And I'm loathe to manage a VM. PaaS is my sugar bear.

My eyes lit up when I saw this. We're an azure shop, but I'm not afraid to use AWS for limited cases. Then I saw - $.30/hr (so, $214/mo). Really? REALLY?

Wouldn't it be comically easy to just the add SFTP as a protocol option for S3? Why does this need a dedicated VM to run it? (Yes, I know this is PaaS and you don't manage the VM, but they're essentially pricing it that way)

HIPAA compliance, even on AWS is extremely expensive. I believe the best vendor to get HIPAA (someone correct me if I'm wrong) is to go with Google Cloud. Last time I checked did not charge any extra for HIPAA BAA signing.

Edit: I stand corrected on this, AWS no longer requires dedicated hardware for BAA HIPAA: Sorry I didn't look this up, I had old information.

https://aws.amazon.com/blogs/apn/aws-hipaa-program-update-re...

AWS requires you to use single tenancy hardware to be covered by that BAA.

OH I stand corrected:

https://aws.amazon.com/blogs/apn/aws-hipaa-program-update-re...

Couldn't you use an Azure webapp? It has FTP and user management, though not as granular as this. You should be able to use IAM to only allow the user access to that one web app and they can set their own username/password. Not quite as simple as sending out creds from one interface, but it's an option. Not sure how many users you have, obviously if you had tons and tons of users it would become a choir, but if it's just a few users I'm thinking a webapp could handle that.

Alternatively, if trusted users enough, you could use an Azure blob and use CloudBerry. That one is probably not HIPPA compliant though.

I don't even know if this new AWS SFTP plan is HIPPA compliant, don't you have to have a log of file check in/outs? And user login logs?

For companies, $214/mo isn'tmuch if it makes an admin's life easier.

At the same time, Amazon isn't going to price it so it's attractive to everybody, because it sounds like they'd rather people not use it if possible. Sounds sensible to me, legacy stuff is always going to cost you one way or another.

People spend millions on stupid stuff like this. The money saved on audit response would justify the service alone.

Frankly, Amazon let a lot of revenue go here — I could think of a few orgs that wouldn’t skip a beat about spending 100x.

I don't know about comically easy, but should be doable to have some sort of shared frontend.

$214 (even $100) is a really beefy vm, though. I wonder whats provisioned and why.

Nothing beefy, probably, but as for HIPAA compliance, AFAIK, you need to sign several specific contracts with your provider and blahblahblah, probably they're just billing you for the incovenience and for having the HIPAA seal.

"I'm HIPAA, the rest aren't, just pay for it".

You have to sign a single business associate agreement (BAA) depending on the nature of the business you are working with. These are usually boilerplate contracts around 2-3 pages long and full of legalese.

It is uncommon for someone to charge you for signing a BAA. It is very common to tie these plans into enterprise only pricing. This is terrible because it adds unnecessary costs to the medical system (which get passed onto consumers) and it completely shuts out smaller players from entering the marketplace. (Cue me side eyeing every single error tracking software SaaS currently on the market who wants to start at 5K a year for their 'small' plan - get real guys)

Yep, I was checking the DICOM (medical images) Viewer plugin for Box the other day and, yeah, they don't charge you for the BAA, but you're required to get an Enterprise or Elite plan, which price isn't even listed, and probably on the thousands:

"Pricing for Box Enterprise or Elite plans as well as the DICOM Viewer additional seat surcharge can be handled by our Box sales team once we know how many seats your are looking for across your company and what types of collaboration use cases you need. "

> and for having the HIPAA seal.

“HIPAA seals”, which several vendors offer, are mostly BS and have no actual meaning under HIPAA.

I don't think any big vendor has them (Google's HIPAA page explicitly notes that no certifications are recognized by the government.)

A tonne of our (enterprise-ey) customers had such trouble trying to integrate into our S3 flow that we started launching VPS for each that abstracted it away into simple SFTP upload/download, which they were used to.

Although this is much more expensive than Lightsail, the man hours saved will make it worthwhile.

Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets.

And each? Surely chrooting users would let you consolidate all of those servers into one (or one cluster for HA I suppose).

> Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets.

Think about the operational costs: someone needs to manage keys, logging, security updates, when S3FS coughs a lung and hangs you need to catch that problem and remount it to restore service, etc. This service reuses the existing authentication systems so you don't need to spend time configuring and managing integration with your customers’ LDAP/AD infrastructure, etc. If you deal with anything which hits PCI, HIPAA, etc. you need to be able to certify that your custom design meets those requirements as well.

That's not to say you can't do it yourself but for many places there's a fairly significant amount of work where the cost of doing it yourself is greater than 5+ years of managed service costs.

Exactly this. If sticker cost is your leading factor then these kinds of services can seem crazy, but when you factor in the real cost of self-hosting then it quickly becomes a no-brainer.

We're more interested in what happens when things break (and who's responsibility it is) than minor cost savings in calm waters.

We use Cloudgates for our FTP/SFTP to S3 interface: https://cloudgates.net/pricing

They're cheap, stable and dead simple to set up. This offering from AWS looks attractive, but at $.30/hour for the server makes it $219/mo vs $25/mo.

edit: just a satisfied customer

So, how do I make sure my connection isn't MITM-ed?There is no server host key anywhere to compare. No CA certificate support. Doesn't look like ed25519 is supported either.

Somehow people don't use self-signed certificates all over the web but for sftp it's "fine" apparently.

You should try the mirror sub-system of lftp [1]. It can replicate rsync behavior on a chroot sftp server. No idea if that works on Amazon, but I use it all the time on my own chroot sftp servers.

[1] - https://tinyvpn.org/sftp/#lftp

lftp is fantastic. The mirror function has a “reverse mode” too

For regular tasks you could also look at “rclone” which is like rsync in many ways but can upload to s3, backblaze b2, sftp and any more directly. Without remote support.

So, I spent a couple days a few months ago building exactly this on an ec-2 instance. I have an SFTP service running on an Ubuntu box, it has jailed homes for users, it's ssh-key-only, uses s3fs to persist things to the correct buckets, etc.

My only problem with the managed service (which I'd LOVE to switch to tbh) is I can't for the life of me get it to actually connect and upload a file. I suspect I'm doing something wrong in IAM, but the tutorials suck and it looks like IAM isn't even ready for this service yet. I can get a user authenticated, but it's like it's trying to figure out where "home" is and crapping out, connection closed. Nothing helpful in the verbose output, either. Bummer.

Ah this would've been so useful 18 months ago. I had to spend MONTHS to convince a vendor (government) to use S3 to upload (keybase-encrypted) files instead of SFTP.

And they finally budged. This would've been so much easier.

Here's an example:

https://github.com/moriyoshi/s3-sftp-proxy

No FUSE. Pure Go so it's low on resource usage and high in platform compatibility. No OpenSSH. No screwing around with Linux users or whatever. Just a single declarative configuration file. You can run this baby in a Docker container with some adjustments to the host if you want this on port 22.

I had to sourcegraph GitHub a bit to find this thing. SEO is so bad on this implementation. I don't know why.

We ended up implementing a REST API endpoint for SFTP to provide an easy way for web apps to transfer content without having to speak the FTP protocol: https://kloudless.com/products/file-storage/

I can see this being valuable for apps to get user content into S3 more efficiently from the server-side rather than funneling it through hosted servers. The one caveat is programmatic user management, which I'm sure is possible.

> SFTP (Secure File Transfer Protocol)

It's SSH File Transfer Protocol. When you say Secure File Transfer Protocol many people think about FTP over SSL if you don't emphasize it's about SSH.

> many people think about FTP over SSL if you don't emphasize it's about SSH

Huh? Sure there's always potential for confusion but every time I heard anything about FTP over SSL (which no one seems to actually use) it's been called "FTPS"

https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol

In computing, the SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream.

It seems there are no web-hooks / callbacks, so you don't get notified when a new file is uploaded (or someone downloads a file).

Another issue is that if your have to support a partner with SFTP data transfer requirements you may have to support one with FTP/FTPS requirements as well. At this point you will have to go to a dedicated FTP server (or outsource it to another company) anyway, and AWS SFTP service will be redundant in this scheme.

You can set a lambda to be triggered on a file upload, at work we do this for creating reports.

Lambda dumps mongo data to an s3 bucket which triggers another lambda to create a csv.

It's S3, so you can use Lambda, it says so in the article.

> You can write AWS Lambda functions to to build an “intelligent” FTP site that processes incoming files as soon as they are uploaded, query the files in situ using Amazon Athena, and easily connect to your existing data ingestion process.

We use a number of AWS LightSail servers for SFTP today, which mostly sit idle.

Will definitely adopt this!

The AWS pricing page for this service says it costs about $225/month for a lightly used instance. I implemented the same kind of thing on AWS using a nano-sized instance for about $10/month. The instance is managed with an Ansible Role for automated SFTP server management. I connected it with an off-the-shelf AWS Lamda function which listens for S3 PUT events and copies files to the SFTP server as needed.

My solution took a little more human-time to setup than the AWS service might, but once setup, it saves about $200/month.

The alternative is developer time. Nothing about managed services is ever less expensive if you don’t account for developer/Devops/netops time saved.

A small company has even more of reason to want as many managed services as possible. You can avoid hiring netops if you both have a third party managed service provider to manage your network and you have developers/architects who know enough to fill in the gaps.

On the other hand, netops staff costs are a lot less... is liquid the right word?

Yes, $200/month is probably not any more than a couple hours/month of even a very lowly paid developer or ops person, once you account for benefits and overhead.

But once you needed to hire that person for any reason... their annual salary is already on the books. Giving them more work to do doesn't affect your budget. But another $2400 a year might. Yeah, if you can avoid hiring that person _at all_... but you probably had some reason you did have to hire a person or three already, and now you've got them.

The actual experience of working in a small under-resourced organization, in my experience, often looks like this.

That’s why you don’t hire them at all. You use an MSP. Even if you do need someone on prem, the simpler you make your infrastructure, the less skilled your netops person has to be. You can hire someone who basically is a help desk person.

When that one netops person leaves, it usually falls on the developers to manage it.

> Nothing about managed services is ever less expensive if you don’t account for developer/Devops/netops time saved.

I literally can't parse this.

Let me try again.

Baremetal vs Cloud hosting -> resource for resource baremetal will almost always end up being cheaper.

The only way you save money on managed services is the cost of management. Meaning every hour that someone doesn't have to spend maintaining infrastructure is a cost savings to the business. Every minute saved by allowing someone else to do the "undifferentiated heavy lifting" is money saved.

Very interesting considering that it enables to make legacy applications work with S3 basically.

The price is quite high for small projects though: $0.30/hour > $216/m > $2592/y

I'd love to see something for plain vanilla FTP.

Many enterprises still use it, would love to see AWS support that as well.

somewhat late to the commenting party but apparently WinSCP can communicate with S3:

https://winscp.net/eng/docs/guide_amazon_s3

> New services rarely launch with CF supportYeah I'm slowly learning that.

Thanks for the tip, at least it's a step in the right direction.

If the file is PUT in a S3 bucket via SFTP, you can trigger a lambda that run CF template.

But that seems kind of Rube Goldbergish. Why not make the change to the file, push it to Git and use CodePipeline with lambda?

1tb for 40,- dollar?

What? Did i just became stupid?