Compress aws files on s3 upload

[Animii]

Summary

Compress state file uploads before storing them in S3.

This change:

• compresses state uploads when SST_STATE_COMPRESS is enabled
• keeps reads backward compatible with both plain JSON and already-compressed state objects

Motivation

Large state files can generate significant S3 upload traffic during deploys because SST writes the state multiple times over the course of a run.

Compressing the state object reduces transfer size substantially while keeping the existing S3 key layout and migration path simple.

Behavior

• SST_STATE_COMPRESS defaults to enabled
• if unset, SST writes compressed state objects with Content-Encoding: gzip
• if SST_STATE_COMPRESS=false or 0, SST writes plain JSON state objects
• reads support both formats regardless of the current flag value

This means existing deployments can migrate in place on the next write, and users can safely toggle compression on or off.
I can remove the flag if is not needed or disable it by default.

Implementation

In pkg/project/provider/aws.go:

• getData() now detects Content-Encoding: gzip and decompresses on read
• putData() now prepares either compressed or plain upload bodies depending on SST_STATE_COMPRESS
• helper functions were extracted to make the behavior easier to test

Tests

Added unit tests covering:

• compressed upload preparation
• uncompressed upload preparation
• backward-compatible decoding of both formats
• invalid gzip input rejection

Benchmark

I benchmarked it on our company project on an already deployed app with ~6900 resources and a ~64 mb state file with a 120 Mbit/s connection.

Results

Uncompressed:
Duration: 4m21s
Total uploads: 14
Toal MB uploaded: 882.08 MB

Compressed:
Duration: 4m26s
Total uploads: 50
Total MB uploaded(compressed): 165.18 MB
Total MB uploaded if uncompressed: 3149.43 MB
Compression ratio is ~5%(64 MB -> 3.4 MB) for the state file

We save a lot of bandwidth and because uploads don't take as much time, SST can upload the state file more frequently.

[vimtor]

thanks for your contribution @Animii

have you tried with PULUMI_DIY_BACKEND_GZIP? not sure how compatible it is with the current sst setup but it should probably work

[Animii]

Yes. With the PULUMI_DIY_BACKEND_GZIP the local files are compressed too, but SST assumes non compressed files.
For example:

sst/pkg/project/workdir.go

Line 119 in a856be1

path := filepath.Join(appDir, fmt.Sprintf("%v.json", w.project.app.Stage))

With the PULUMI_DIY_BACKEND_GZIP flag enabled SST can't find these files and crashes.

It would take more invasive work on how SST handles state files to make it compatible(fully delegate state file handling to Pulumi?!).
I think the current approach has a cleaner separation since it only cares about S3 uploads.

[Animii]

@vimtor Even if we make SST compatible with the files that Pulumi creates, that would not compress files that SST creates and uploads.

[vimtor]

@Animii makes sense, i still need to figure out if we should be using the s3 option pulumi provides for diy backends

not sure that was available when this feature was built into sst