Commit Graph

90 Commits (9b84c58b3bc7679a7ec54f096a60641251eb9e31)

Author SHA1 Message Date
Frostebite 962603b7b3 refactor(container-hook-service): improve AWS hook inclusion logic based on provider strategy and credentials; update binary files 2025-09-04 15:35:34 +01:00
Frostebite 8acf3ccca3 refactor(build-automation): enhance containerized workflow handling and log management; update builder path logic based on provider strategy 2025-09-04 01:29:41 +01:00
Frostebite ec93ad51d9 chore(format): prettier/eslint fix for build-automation-workflow; guard local provider steps 2025-09-04 00:24:28 +01:00
Frostebite c3e0ee6d1a ci(aws): echo CACHE_KEY during setup to ensure e2e sees cache key in logs; tests: retained workspace AWS assertion (#381) 2025-09-04 00:03:42 +01:00
Frostebite f2dbcdf433 style(remote-client): satisfy eslint lines-around-comment; tests: log cache key for retained workspace (#379) 2025-09-03 22:40:52 +01:00
Frostebite c8f881a385 tests: assert BuildSucceeded; skip S3 locally; AWS describeTasks backoff; lint/format fixes 2025-09-03 20:49:52 +01:00
Frostebite eb8b92cda1 Update log output handling in FollowLogStreamService to always append log lines for test assertions 2025-09-03 18:32:27 +01:00
Frostebite 0650d1de5c fix 2025-08-28 06:48:50 +01:00
Frostebite 6e13713bb2 Merge branch 'main' into cloud-runner-develop
# Conflicts:
#	dist/index.js
#	dist/index.js.map
#	jest.config.js
#	yarn.lock
2025-08-18 20:54:05 +01:00
Frostebite fa6440db27
Merge branch 'main' into codex/use-aws-sdk-for-workspace-locking 2025-08-07 17:00:03 +01:00
Frostebite c6c8236152
fix: mock github checks in tests (#724)
* fix: load fetch polyfill before tests

* refactor: extract cloud runner test helpers

* fix: load fetch polyfill before tests
2025-08-06 06:07:52 +01:00
Frostebite 5b34e4df94 fix: lazily initialize S3 client 2025-08-05 01:48:46 +01:00
Frostebite 12e5985cf8 refactor: use AWS SDK for workspace locks 2025-08-04 20:35:23 +01:00
Frostebite a0833df59e fix 2025-06-30 15:46:52 +01:00
Frostebite b662a6fa0e Refactor URL configuration in RemoteClient for token-based authentication
- Updated comments for clarity regarding the purpose of URL configuration changes.
- Simplified the git configuration commands by removing redundant lines while maintaining functionality for HTTPS token-based authentication.
- This change enhances the readability and maintainability of the RemoteClient class's git setup process.
2025-06-10 20:09:34 +01:00
Frostebite 36503e30c0 Update git configuration commands in RemoteClient to ensure robust URL unsetting
- Modified the git configuration commands to append '|| true' to prevent errors if the specified URLs do not exist.
- This change enhances the reliability of the URL clearing process in the RemoteClient class, ensuring smoother execution during token-based authentication setups.
2025-06-10 00:52:48 +01:00
Frostebite 1815c3c880 Refactor git configuration for LFS file pulling with token-based authentication
- Enhanced the process of configuring git to use GIT_PRIVATE_TOKEN and GITHUB_TOKEN by clearing existing URL configurations before setting new ones.
- Improved the clarity of the URL replacement commands for better readability and maintainability.
- This change ensures a more robust setup for pulling LFS files in environments requiring token authentication.
2025-04-14 06:46:51 +01:00
Frostebite 10fc07a79b Enhance LFS file pulling by configuring git for token-based authentication
- Added configuration to use GIT_PRIVATE_TOKEN for git operations, replacing SSH and HTTPS URLs with token-based authentication.
- Improved error handling to ensure GIT_PRIVATE_TOKEN availability before attempting to pull LFS files.
- This change streamlines the process of pulling LFS files in environments requiring token authentication.
2025-04-14 01:37:55 +01:00
Frostebite a1f3d9ecd4 Enhance LFS file pulling with token fallback mechanism
- Implemented a primary attempt to pull LFS files using GIT_PRIVATE_TOKEN.
- Added a fallback mechanism to use GITHUB_TOKEN if the initial attempt fails.
- Configured git to replace SSH and HTTPS URLs with token-based authentication for the fallback.
- Improved error handling to log specific failure messages for both token attempts.

This change ensures more robust handling of LFS file retrieval in various authentication scenarios.
2025-04-13 18:49:33 +01:00
Matheus Costa 81ed299e10
Feat/migrate aws sdk v3 (#698)
* chore(cloud-runner): migrate/replace deps aws-sdk v2 to v3

* chore(aws): refactor aws services to support SDK v3

* chore(aws): refactor aws runner to support SDK v3

* chore(aws): update dist

* fix(aws): error handling wrap try/catch to avoid unhandled promise rejections.

* fix(aws): keeping the syntax simpler for arrays
2025-04-10 22:48:14 +02:00
Frostebite e73b48fb38
Cloud runner develop - Stabilizes kubernetes provider (#531)
* fixes

* fixes

* fixes

* fixes

* fixes

* check for startup message in workflows

* check for startup message in workflows

* check for startup message in workflows

* check for startup message in workflows

* check for startup message in workflows

* check for startup message in workflows

* Update cloud-runner-ci-pipeline.yml

* Update cloud-runner-ci-pipeline.yml

* no storage class specified

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* updates

* log file path

* latest develop

* log file path

* log file path

* Update package.json

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* log file path

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* stream logs through standard input and new remote client cli command

* update pipeline to use k3s

* version: 'latest'

* fixes

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* disable aws pipe for now

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* push k8s logs to LOG SERVICE IP

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* tests

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* podname logs for log service

* hashed logs

* hashed logs

* hashed logs

* hashed logs

* hashed logs

* hashed logs

* no wait, just repeat logs

* no wait, just repeat logs

* remove typo - double await

* test fix - kubernetes - name typo in github yaml

* test fix - kubernetes - name typo in github yaml

* check missing log file

* check missing log file

* Push to steam test

* Push to steam test

* Fix path

* k8s reliable log hashing

* k8s reliable log hashing

* k8s reliable log hashing

* hashed logging k8s

* hashed logging k8s

* hashed logging k8s

* hashed logging k8s

* hashed logging k8s

* hashed logging k8s

* Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line

* Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line

* Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line

* Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line

* Include log chunk when task runner sees log update, clarify if we can pull logs from same line or next line

* Fix exit flow for k8s job

* hash comparison logging for log complete in k8s flow

* Interrupt k8s logs when logs found

* cleanup async parameter

* cleanup async parameter

* cleanup async parameter

* fixes

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2024-02-06 23:46:31 +00:00
Andrew Kahr bbd713b05a
Fix pro activation (#602)
- Only randomize uuid for personal licenses
- Add warning annotation for license activation retries
- add `engineExitCode` output
- repo/code cleanup
2023-11-27 23:24:58 -08:00
Webber Takken a073719c29
BREAKING (potentially) - Bump docker version (Ubuntu 22.04 LTS) (#556)
* fix: resolution errors and vulnerability

* feat: bump (major) docker image rolling tag

* chore: bump major version

* fix: up workflow node to lts

* fix: conventions
2023-08-22 10:03:32 +02:00
Frostebite 7abb3a409d
Cloud runner develop - latest fixes (#524)
Cloud runner develop - latest fixes (#524)
2023-03-27 12:14:23 +01:00
AndrewKahr ef38f5a88a
Code cleanup (#511)
* Enable noImplicitAny
Add types to all implicit any variables
Bump target to ES2020 for recent language features (optional chaining)
Code cleanup
Add debug configuration for vscode
Remove autorun flag from jest to remove warning
Bump packages to fix dependency version mismatch warning
Changed @arkweid/lefthook to @evilmartians/lefthook as @arkweid/lefthook has been deprecated in favor of @evilmartians/lefthook
Added concurrency groups to integrity check and build workflows. New commits to branches will cancel superseded runs on the same branch/pr
Update imports to not use require syntax
Use node packages (ie node:fs rather than fs)
AndroidVersionCode is now a string rather than a number as it gets converted to a string when passed out of the system
Reduce timeout for windows builds
Remove 2020.1.17f1 from windows builds due to repeated license activation errors
Update naming scheme of workflows for consistency
Update build names so target platform and unity version aren't cut off by github actions UI

* Add exclude to test matrix for 2022.2 on android until Unity bug is fixed

---------

Co-authored-by: AndrewKahr <AndrewKahr@users.noreply.github.com>
2023-03-03 16:25:40 -08:00
Frostebite 28147e5e1b Revert "triggerWorkflowOnComplete param for cloud runner"
This reverts commit 00c5685d03.
2023-02-05 00:22:32 +00:00
Frostebite 00c5685d03 triggerWorkflowOnComplete param for cloud runner 2023-02-05 00:16:42 +00:00
Frostebite e334dc785a
Cloud runner develop - better parameterization of s3 usage, improved async workflow and GC, github checks early integration (#479)
* custom steps may leave value undefined, will be pulled from env vars

* custom steps may leave value undefined, will be pulled from env vars

* custom steps may leave value undefined, will be pulled from env vars

* add 3 new premade steps, steam-deploy-client, steam-deploy-project, aws-s3-pull-build

* fix

* fix

* fix

* continue building async-workflow support

* test checks

* test checks

* test checks

* move github checks within build workflow

* async workflow test

* async workflow test

* async workflow test

* async workflow test

* async workflow test

* async workflow test

* async workflow test

* async workflow test for aws only

* async workflow test for aws only

* async workflow test for aws only

* async workflow test for aws only

* cleanup logging

* disable lz4 compression by default

* disable lz4 compression by default

* AWS BASE STACK for tests

* AWS BASE STACK for tests

* AWS BASE STACK for tests

* AWS BASE STACK for tests

* AWS BASE STACK for tests

* AWS BASE STACK for tests

* disable lz4 compression by default

* disable lz4 compression by default

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* Update github check with aws log

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* kinesis and subscription filter for logs creation skipped when watchToEnd false

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* cleanup local pipeline, log aws formation

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* async pipeline

* workflow

* workflow

* workflow

* workflow

* workflow

* workflow

* workflow

* workflow

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3

* parameterize s3
2023-01-20 17:40:57 +00:00
Frostebite 2a32a9d870
Fix: retained workspaces must fetch before resetting to needed commit (#476) 2022-11-18 20:43:52 +00:00
Frostebite d6ac850da1
Fix "large package handling" to redirect to shared folder (#473) 2022-11-17 18:59:33 +00:00
Frostebite f300dd27bb
Add Cloud Runner Pipeline to Main (#472)
* fix: k8s error handling

(cherry picked from commit f633a3efb42432a6d2492712aead865a950c8dca)

* include main in main cloud-runner pipeline

(cherry picked from commit a40fbe941bba1ba4593c83c754b37363a969bfe5)
2022-11-08 09:37:16 +00:00
Frostebite 4cca069ebb
Cloud Runner Develop - 1.0 R.C 1 (#437)
Release Candidate changeset 1 - For 1.0 Cloud Runner
2022-11-07 20:41:00 +00:00
simensan 4cb3e593f5
Feature/support for unity licensing server linux (#468)
* Initial support for adding a UNITY_LICENSING_SERVER parameter to build parameters

* Test to figure out what the working directory is of current bash script

* Outputting current directory and using $ACTION_FOLDER

* Add resources folder to mounted docker volumes. Used by activation script to copy over template file for unity licensing server

* use awk instead of sed due to http characters breaking syntax

* mkdir for unity config

* Add -p flag to mkdir so parents are also created if missing

* Initial work on returning floating license when using licensing server

* Checking licensing server first for now, since serial is always set

* Parse and save acquired floating license for use for returning after build

* Clean up duplicate commands in activate.sh

* Fixed running string as command, use it as input instead

* Fixed cloud runner tests failing when using a ssh remote.

* Clean up of test files and unnecessary logging

* Moved process of generating services-config.json file from platform specific activate scripts to typescript

* Fixed path
2022-10-22 18:55:58 +02:00
Frostebite f77696efae
Cloud runner develop v0.1 (#395)
* Correct aws logs link

* Correct aws logs link

* better aws cli commands and better cleanup for aws

* better aws cli commands and better cleanup for aws

* improved garbage collection cli options

* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup

* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup

* fix issue #393

* Extract follow log stream service

* consolidate into one pipeline file

* consolidate into one pipeline file
2022-05-05 00:25:17 +01:00
Frostebite 8abce48a48
Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt

* cloud-runner-system prefix changed to cloud-runner

* Update cloud-runner-aws-pipeline.yml

* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default

* github ephemeral pipeline support

* github ephemeral pipeline support

* Merge remote-tracking branch 'origin/main' into cloud-runner-develop

# Conflicts:
#	dist/index.js.map
#	src/model/cloud-runner/providers/aws/aws-task-runner.ts
#	src/model/cloud-runner/providers/aws/index.ts

* garbage collection

* garbage collection

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* download runner each time

* download runner each time

* download runner each time

* garbage collect all older than 1d as part of cleanup

* download runner each time

* number container cpu and memory for aws

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* Skip printing size unless cloudRunnerIntegrationTests is true

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* better aws commands

* better aws commands

* parse number for cloud formation template

* remove container resource defaults from actions yaml

* remove container resource defaults from actions yaml

* skip all input readers when cloud runner is local

* prefer fs/promises

* actually set aws cloud runner step as failure if unity build fails

* default to 3gb of ram - webgl fails on 2
2022-04-22 00:47:45 +01:00
Webber Takken 5ae03dfef6
Streamline code styles (#384)
* feat: streamline code styles

* feat: spacing for comments and return statements

* chore: enforce camelcase

* fix: remove npm lock file

* fix: add integrity test

* fix: remove logfile

* chore: update node in test workflow
2022-04-12 00:43:41 +02:00
Frostebite a61c02481f
Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt
2022-04-11 00:00:37 +01:00
Webber Takken 441be81543
Switch to version 1 images (#374)
* feat: upgrade to images of version 1 (rolling tag)

* chore: indicate what needs to move out of the input class
2022-04-03 17:59:14 +02:00
Frostebite d975f3b80f
Minor tweaks and corrections for cloud runner pipeline and branch parameter (#351)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo
2022-03-04 19:42:24 +00:00
Frostebite 03ae77dc7c
Cloud Runner v2 (#310) 2022-02-01 02:31:20 +00:00