Commit Graph

132 Commits (0d4dcfdd334b43fd6e3a10dc703bfe22679aa86a)

Author SHA1 Message Date
Frostebite 87d17e4efe retained workspace for k8s 2022-10-17 22:11:50 +01:00
Frostebite e4b6a149f2 retained workspace for k8s 2022-10-17 20:59:40 +01:00
Frostebite bc8208b484 cleanup 2022-10-17 17:36:22 +01:00
Frostebite 20a967df3b cleanup 2022-10-16 02:47:58 +01:00
Frostebite 3046c521a8 cleanup 2022-10-14 16:35:11 +01:00
Frostebite b5de40caa2 cleanup 2022-10-14 01:04:34 +01:00
Frostebite f727a3c06c cleanup 2022-10-14 00:36:55 +01:00
Frostebite 37bc098549 cleanup 2022-10-13 21:54:37 +01:00
Frostebite 4c4e25f21a Prebuilt steps 2022-10-13 18:54:25 +01:00
Frostebite 9bc4650b83 Custom step file support 2022-10-08 04:42:30 +01:00
Frostebite 0147dac3c2 Custom step file support 2022-10-08 04:23:24 +01:00
Frostebite 48f3a4812c custom hook files and test 2022-10-06 22:05:42 +01:00
Frostebite 2b4295a004 Fix: check for retained workspace 2022-10-06 01:37:55 +01:00
Frostebite 7d404398cc Fix: check for retained workspace 2022-10-05 20:52:32 +01:00
Frostebite 954066b224 Copy all of data folder to docker volume to enable local-docker retained workspace 2022-10-05 20:38:40 +01:00
Frostebite 9253b5e21c Skip all locking actions test as we now have two useful test flows 2022-10-05 20:28:46 +01:00
Frostebite 5207bf8ae4 Skip all locking actions test as we now have two useful test flows 2022-10-05 20:09:51 +01:00
Frostebite 2bfe623bb7 Better locking 2022-09-30 01:31:01 +01:00
Frostebite 3a5400080b Better locking 2022-09-29 21:30:34 +01:00
Frostebite 026f8f20a4 K8s log and stringify error for k8s cleanup 2022-09-29 19:57:34 +01:00
Frostebite f68dc13959 K8s log and stringify error for k8s cleanup 2022-09-29 17:59:45 +01:00
Frostebite 2342501554 K8s log and stringify error for k8s cleanup 2022-09-29 17:35:45 +01:00
Frostebite 0f02fbfd32 K8s log and stringify error for k8s cleanup 2022-09-29 16:53:46 +01:00
Frostebite 1b09122e35 remove logs 2022-09-28 03:17:20 +01:00
Frostebite 8a6bf7d142 Copy cloud runner cache into github workspace directory 2022-09-28 02:11:13 +01:00
Frostebite 45d4191982 Copy cloud runner cache into github workspace directory 2022-09-28 02:07:31 +01:00
Frostebite f5e6ab28c1 Copy cloud runner cache into github workspace directory 2022-09-28 01:59:42 +01:00
Frostebite 9921b65ebc Copy cloud runner cache into github workspace directory 2022-09-28 01:48:09 +01:00
Frostebite a265b507a6 Copy cloud runner cache into github workspace directory 2022-09-28 01:10:53 +01:00
Frostebite d950f8b6ef Copy cloud runner cache into github workspace directory 2022-09-28 01:09:18 +01:00
Frostebite 48149e70a9 Copy cloud runner cache into github workspace directory 2022-09-28 01:03:15 +01:00
Frostebite daa844d053 Do not serialize custom job parameter 2022-09-28 00:23:16 +01:00
Frostebite 5445231c42 Do not serialize custom job parameter 2022-09-28 00:01:09 +01:00
Frostebite 6170f2320a Do not serialize custom job parameter 2022-09-27 23:59:01 +01:00
Frostebite 53ebd09102 Do not serialize custom job parameter 2022-09-27 23:36:40 +01:00
Frostebite 8a0b5e903b Do not serialize custom job parameter 2022-09-27 23:17:06 +01:00
Frostebite 98536a48a0 Do not serialize custom job parameter 2022-09-27 23:12:42 +01:00
Frostebite cf95b5e11d Do not serialize custom job parameter 2022-09-27 23:05:40 +01:00
Frostebite 4e2b47d4cd cache test should explicitly check cache key from build 1 is included in results log 2022-09-25 03:17:11 +01:00
Frostebite 095539b0e2 cache test should explicitly check cache key from build 1 is included in results log 2022-09-22 23:38:25 +01:00
Frostebite 870b6eabf8 cache test should explicitly check cache key from build 1 is included in results log 2022-09-22 22:48:23 +01:00
Frostebite 5353abd16e better named tests and log local-docker params 2022-09-22 22:42:17 +01:00
Frostebite ffc984be4b better named tests and log local-docker params 2022-09-22 22:36:33 +01:00
Frostebite a03e4c21d4 better named tests and log local-docker params 2022-09-22 22:05:05 +01:00
Frostebite 94851b231b better named tests and log local-docker params 2022-09-22 22:00:57 +01:00
Frostebite 89993f452b better named tests and log local-docker params 2022-09-21 22:00:42 +01:00
Frostebite c35fe5e6a6 better named tests and log local-docker params 2022-09-21 21:38:14 +01:00
Frostebite 6de6e1e1a2 better named tests and log local-docker params 2022-09-21 16:40:22 +01:00
Frostebite 62949bc520 better named tests and log local-docker params 2022-09-21 16:33:12 +01:00
Frostebite 04c03528ea better named tests and log local-docker params 2022-09-21 16:24:09 +01:00
Frostebite ce06645084 better named tests and log local-docker params 2022-09-21 15:17:40 +01:00
Frostebite f53f39bf3f better named tests and log local-docker params 2022-09-21 14:09:39 +01:00
Frostebite c3e5478a18 better named tests and log local-docker params 2022-09-21 13:54:17 +01:00
Frostebite f029159324 better named tests and log local-docker params 2022-09-21 13:39:00 +01:00
Frostebite 3377ebfda5 better named tests and log local-docker params 2022-09-21 13:21:13 +01:00
Frostebite 7c332a36fb better named tests and log local-docker params 2022-09-21 03:22:26 +01:00
Frostebite 22d6a89d26 better named tests and log local-docker params 2022-09-21 02:56:48 +01:00
Frostebite c9678c29b6 better named tests and log local-docker params 2022-09-21 01:11:03 +01:00
Frostebite 16c6f661ee better named tests and log local-docker params 2022-09-20 23:14:08 +01:00
Frostebite 9680daecd9 better named tests and log local-docker params 2022-09-20 22:53:00 +01:00
Frostebite 0cc6270b68 better named tests and log local-docker params 2022-09-20 22:23:42 +01:00
Frostebite fe9da282a7 better named tests and log local-docker params 2022-09-20 22:17:26 +01:00
Frostebite 1771726c36 get output from local-docker 2022-09-19 02:12:28 +01:00
Frostebite aa1f3ccc51 get output from local-docker 2022-09-19 01:26:33 +01:00
Frostebite 5a46718255 get output from local-docker 2022-09-19 01:21:56 +01:00
Frostebite 14a832395b get output from local-docker 2022-09-19 01:13:27 +01:00
Frostebite 20476c3ccc local-docker provider use docker with commands 2022-09-17 09:14:43 +01:00
Frostebite 8031bac31f local-docker provider use docker with commands 2022-09-17 09:05:43 +01:00
Frostebite 307bea209b local-docker provider use docker with commands 2022-09-17 08:43:09 +01:00
Frostebite 92bcdf5e96 local-docker provider use docker with commands 2022-09-17 08:37:57 +01:00
Frostebite f226bef5af
Cloud runner preview (#419) 2022-09-16 19:48:40 +01:00
Frostebite 8bd51be73f improve cleanup to not delete in use task clusters 2022-05-14 17:29:09 +01:00
Frostebite 16b81cc6c8 Extract follow log stream service 2022-05-02 09:11:29 +01:00
Frostebite 6668e1133e Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup 2022-05-01 19:34:48 +01:00
Frostebite a47d096f5c improved garbage collection cli options 2022-05-01 19:32:59 +01:00
Frostebite fc3724600e better aws cli commands and better cleanup for aws 2022-05-01 08:29:04 +01:00
Frostebite 3bd297d1c8 better aws cli commands and better cleanup for aws 2022-05-01 08:24:55 +01:00
Frostebite a184b7abae Correct aws logs link 2022-05-01 05:43:17 +01:00
Frostebite 483bcf9dd4 Correct aws logs link 2022-05-01 05:40:56 +01:00
Frostebite 8abce48a48
Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt

* cloud-runner-system prefix changed to cloud-runner

* Update cloud-runner-aws-pipeline.yml

* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default

* github ephemeral pipeline support

* github ephemeral pipeline support

* Merge remote-tracking branch 'origin/main' into cloud-runner-develop

# Conflicts:
#	dist/index.js.map
#	src/model/cloud-runner/providers/aws/aws-task-runner.ts
#	src/model/cloud-runner/providers/aws/index.ts

* garbage collection

* garbage collection

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* download runner each time

* download runner each time

* download runner each time

* garbage collect all older than 1d as part of cleanup

* download runner each time

* number container cpu and memory for aws

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* Skip printing size unless cloudRunnerIntegrationTests is true

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* better aws commands

* better aws commands

* parse number for cloud formation template

* remove container resource defaults from actions yaml

* remove container resource defaults from actions yaml

* skip all input readers when cloud runner is local

* prefer fs/promises

* actually set aws cloud runner step as failure if unity build fails

* default to 3gb of ram - webgl fails on 2
2022-04-22 00:47:45 +01:00
Webber Takken 5ae03dfef6
Streamline code styles (#384)
* feat: streamline code styles

* feat: spacing for comments and return statements

* chore: enforce camelcase

* fix: remove npm lock file

* fix: add integrity test

* fix: remove logfile

* chore: update node in test workflow
2022-04-12 00:43:41 +02:00
Frostebite a61c02481f
Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt
2022-04-11 00:00:37 +01:00