Commit Graph

48 Commits (086194999eef2f08ffb727b2e3578a055223796b)

Author SHA1 Message Date
Frostebite e4b6a149f2 retained workspace for k8s 2022-10-17 20:59:40 +01:00
Frostebite a320cf893c cleanup 2022-10-14 20:04:14 +01:00
Frostebite 7fd983757d cleanup 2022-10-14 20:01:51 +01:00
Frostebite a2b26a7fa2 cleanup 2022-10-14 18:07:23 +01:00
Frostebite 2b24bb8882 cleanup 2022-10-14 17:54:34 +01:00
Frostebite 395a2c8204 cleanup 2022-10-14 17:38:44 +01:00
Frostebite a5497cd7c7 cleanup 2022-10-14 17:24:47 +01:00
Frostebite f9100782b8 Cleanup tree usage 2022-10-13 19:21:40 +01:00
Frostebite 4c4e25f21a Prebuilt steps 2022-10-13 18:54:25 +01:00
Frostebite 1e2679d2d6 Custom step file support 2022-10-13 17:14:00 +01:00
Frostebite 3a2aedaf9e Custom step file support 2022-10-13 16:53:46 +01:00
Frostebite 6f985fa22d Custom step file support 2022-10-13 16:33:52 +01:00
Frostebite 61a036f034 Custom step file support 2022-10-13 16:33:05 +01:00
Frostebite 8214abc1a9 Custom step file support 2022-10-13 16:09:41 +01:00
Frostebite 0809b30d22 Custom step file support 2022-10-13 15:57:01 +01:00
Frostebite f0723fffdd Custom step file support 2022-10-13 15:42:21 +01:00
Frostebite 76bab5b221 Custom step file support 2022-10-13 15:41:04 +01:00
Frostebite 237b1df2c9 Custom step file support 2022-10-11 14:58:09 +01:00
Frostebite ac47f82794 Custom step file support 2022-10-10 22:05:03 +01:00
Frostebite b0b5041c22 Custom step file support 2022-10-10 22:04:39 +01:00
Frostebite a0891bdc4a Custom step file support 2022-10-10 21:58:25 +01:00
Frostebite f7932ef20a Custom step file support 2022-10-10 21:42:53 +01:00
Frostebite 65d9371b05 Custom step file support 2022-10-10 21:36:02 +01:00
Frostebite 4e0e80b838 Custom step file support 2022-10-09 05:04:42 +01:00
Frostebite 81be8b8038 Custom step file support 2022-10-09 00:39:21 +01:00
Frostebite 36720f7f51 Custom step file support 2022-10-08 17:16:34 +01:00
Frostebite 893209cd4f Custom step file support 2022-10-08 01:12:39 +01:00
Frostebite b5c63e451d Custom step file support 2022-10-08 00:32:52 +01:00
Frostebite 7fcd51349b
Cloud Runner 1.0 (#459)
* Fix: post build caching, use linux path conversion

* Fix: post build caching via CLI

* Fix: post build caching via CLI

* Fix: post build caching via CLI

* Fix: post build caching via CLI

* Fix: post build caching via CLI

* Log if retained workspace option is present for testing

* Log if retained workspace option is present for testing

* Use retained workspace :O

* Use retained workspace :O

* Use retained workspace :O

* Use retained workspace :O

* Ignore garbage creating lock actions in test for now

* Lock workspace when using Get or Create Locked Workspace

* Lock workspace before creating workspace file to allow for an unblockable creation sequence with guarenteed lock for the original creator

* intuitive locking logs from the most important flow

* test naming

* consider lock folders without workspace file locked

* Use cache key to segment lock folders

* Use cache key to segment lock folders

* Use cache key to segment lock folders

* Use cache key to segment lock folders

* Use cache key to segment lock folders

* Skip all locking actions test as we now have two useful test flows

* Skip all locking actions test as we now have two useful test flows

* Skip all locking actions test as we now have two useful test flows

* Copy all of data folder to docker volume to enable local-docker retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Fix: check for retained workspace

* Skip main clone if game repo exists

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* handle cloud runner git sync via sha not only branch

* transfer locked workspace to static CloudRunner field

* transfer locked workspace to static CloudRunner field

* transfer locked workspace to static CloudRunner field

* custom hook files and test

* custom hook files and test

* custom hook files and test
2022-10-06 20:42:33 +01:00
Frostebite b41839b2f5 Fix: post build caching, use linux path conversion 2022-10-02 19:00:13 +01:00
Frostebite cbde7b54ac Extract post build steps into CLI command 2022-10-02 17:52:38 +01:00
Frostebite aca6752ecd Simple retained workspace check exploration and stubbing post build steps as a CLI command 2022-10-02 16:44:55 +01:00
Frostebite 03aae52afe Only use locking in build workflow if retained workspaces is enabled 2022-10-01 00:40:44 +01:00
Frostebite d4c5218b39 Better locking 2022-09-29 22:03:16 +01:00
Frostebite 1beee459f2 Better locking 2022-09-29 21:38:19 +01:00
Frostebite 3a5400080b Better locking 2022-09-29 21:30:34 +01:00
Frostebite 250d9de02f better named tests and log local-docker params 2022-09-21 04:27:19 +01:00
Frostebite 2721aba2b5 better named tests and log local-docker params 2022-09-21 02:15:07 +01:00
Frostebite 3861aac122 better named tests and log local-docker params 2022-09-20 23:07:37 +01:00
Frostebite ce6a47b3b6 better named tests and log local-docker params 2022-09-20 23:03:04 +01:00
Frostebite 73bad04ed6 tree cache push directory 2022-09-18 23:08:27 +01:00
Frostebite b9608f9169 fix push to cache 2022-09-17 01:09:28 +01:00
Frostebite 01ccc8ee10 fix push to cache 2022-09-16 22:24:23 +01:00
Frostebite f226bef5af
Cloud runner preview (#419) 2022-09-16 19:48:40 +01:00
Frostebite 8abce48a48
Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt

* cloud-runner-system prefix changed to cloud-runner

* Update cloud-runner-aws-pipeline.yml

* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default

* github ephemeral pipeline support

* github ephemeral pipeline support

* Merge remote-tracking branch 'origin/main' into cloud-runner-develop

# Conflicts:
#	dist/index.js.map
#	src/model/cloud-runner/providers/aws/aws-task-runner.ts
#	src/model/cloud-runner/providers/aws/index.ts

* garbage collection

* garbage collection

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* download runner each time

* download runner each time

* download runner each time

* garbage collect all older than 1d as part of cleanup

* download runner each time

* number container cpu and memory for aws

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* Skip printing size unless cloudRunnerIntegrationTests is true

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* better aws commands

* better aws commands

* parse number for cloud formation template

* remove container resource defaults from actions yaml

* remove container resource defaults from actions yaml

* skip all input readers when cloud runner is local

* prefer fs/promises

* actually set aws cloud runner step as failure if unity build fails

* default to 3gb of ram - webgl fails on 2
2022-04-22 00:47:45 +01:00
Webber Takken 5ae03dfef6
Streamline code styles (#384)
* feat: streamline code styles

* feat: spacing for comments and return statements

* chore: enforce camelcase

* fix: remove npm lock file

* fix: add integrity test

* fix: remove logfile

* chore: update node in test workflow
2022-04-12 00:43:41 +02:00
Frostebite a61c02481f
Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt
2022-04-11 00:00:37 +01:00
Frostebite 03ae77dc7c
Cloud Runner v2 (#310) 2022-02-01 02:31:20 +00:00