Commit Graph

512 Commits (77aec2b711d7ae798224b5c4797816ad69467981)

Author SHA1 Message Date
Frostebite 2992fb6115 Remove additional early caching test step in ci pipeline 2022-09-17 07:03:50 +01:00
Frostebite d017aa0d55 Do not include all env vars as secrets! 2022-09-17 06:54:34 +01:00
Frostebite 67696d6253 Do not include all env vars as secrets! 2022-09-17 06:46:59 +01:00
Frostebite 6daf32f531 Do not include all env vars as secrets! 2022-09-17 06:30:51 +01:00
Frostebite a330d4e9a3 better startup and teardown for tests 2022-09-17 05:44:07 +01:00
Frostebite 5bf0ad0367 better startup and teardown for tests 2022-09-17 05:09:36 +01:00
Frostebite 51d46c2cc0 lock put timestamp first 2022-09-17 04:49:44 +01:00
Frostebite cefd469364 lock test 2022-09-17 04:41:41 +01:00
Frostebite 9ed3251455 lock test 2022-09-17 04:38:00 +01:00
Frostebite 54eab9797b Fix s3 command for workspace locking 2022-09-17 04:25:29 +01:00
Frostebite f72739254b Fix s3 command for workspace locking 2022-09-17 04:20:09 +01:00
Frostebite 33eb514ce6 Fix cloud runner test - format secret names for validation 2022-09-17 03:52:50 +01:00
Frostebite c5a37b5086 Fix cloud runner test - format secret names for validation 2022-09-17 03:22:04 +01:00
Frostebite d0b850169b Remove unintended change 2022-09-17 02:18:57 +01:00
Frostebite 7aa9d1ab16 Fix: use project path not repo path to find Packages/manifest.json 2022-09-17 01:50:19 +01:00
Frostebite b964d3100a Stub custom hook+step file parameter and CLI command to run 2022-09-17 01:27:38 +01:00
Frostebite 38b7b8e1f9 VS Code force LF endings 2022-09-17 01:14:29 +01:00
Frostebite b9608f9169 fix push to cache 2022-09-17 01:09:28 +01:00
Frostebite d69f044655 fix push to cache 2022-09-17 01:06:38 +01:00
Frostebite 2f1a7b21fb fix push to cache 2022-09-17 00:57:38 +01:00
Frostebite 3350035d94 Test secrets are serialized correctly 2022-09-17 00:56:22 +01:00
Frostebite 3f69ca579c fix push to cache 2022-09-17 00:23:41 +01:00
Frostebite df240154da Test secrets are serialized correctly 2022-09-16 23:51:19 +01:00
Frostebite 01ccc8ee10 fix push to cache 2022-09-16 22:24:23 +01:00
Frostebite f226bef5af
Cloud runner preview (#419) 2022-09-16 19:48:40 +01:00
Frostebite 17faf202fb
Update cloud-runner-pipeline.yml 2022-06-02 22:32:33 +01:00
Frostebite a4c0e7626d
Update cloud-runner-pipeline.yml 2022-05-19 01:58:33 +01:00
Frostebite 8bd51be73f improve cleanup to not delete in use task clusters 2022-05-14 17:29:09 +01:00
Frostebite 1d2c23fdad Merge remote-tracking branch 'origin/main' into cloud-runner-develop 2022-05-14 17:13:02 +01:00
David Finol 2652cb78a7
Fix mac return license (#397) 2022-05-08 06:18:37 -05:00
Frostebite f77696efae
Cloud runner develop v0.1 (#395)
* Correct aws logs link

* Correct aws logs link

* better aws cli commands and better cleanup for aws

* better aws cli commands and better cleanup for aws

* improved garbage collection cli options

* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup

* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup

* fix issue #393

* Extract follow log stream service

* consolidate into one pipeline file

* consolidate into one pipeline file
2022-05-05 00:25:17 +01:00
Frostebite bb9323399f
Merge branch 'main' into cloud-runner-develop 2022-05-02 18:30:16 +01:00
Khalid.Sani 4556fc4ff1
fix: git private token not being set correctly for https (#394) 2022-05-02 17:04:05 +02:00
Frostebite f9492052d8 consolidate into one pipeline file 2022-05-02 09:21:07 +01:00
Frostebite ef19bdf03a consolidate into one pipeline file 2022-05-02 09:19:34 +01:00
Frostebite 16b81cc6c8 Extract follow log stream service 2022-05-02 09:11:29 +01:00
Frostebite 795ae32499 fix issue #393 2022-05-01 20:56:08 +01:00
Frostebite 76fc76bbcd Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup 2022-05-01 19:50:29 +01:00
Frostebite 6668e1133e Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup 2022-05-01 19:34:48 +01:00
Frostebite a47d096f5c improved garbage collection cli options 2022-05-01 19:32:59 +01:00
Frostebite 606102aae4
Merge branch 'main' into cloud-runner-develop 2022-05-01 18:16:23 +01:00
Frostebite fc3724600e better aws cli commands and better cleanup for aws 2022-05-01 08:29:04 +01:00
Frostebite 3bd297d1c8 better aws cli commands and better cleanup for aws 2022-05-01 08:24:55 +01:00
Frostebite a184b7abae Correct aws logs link 2022-05-01 05:43:17 +01:00
Frostebite 483bcf9dd4 Correct aws logs link 2022-05-01 05:40:56 +01:00
David Finol d066039c26
Fix for 2021.3 (#392)
As a workaround for an error in 2021.3, write logs to file, and display the logs later: https://forum.unity.com/threads/exception-occurred-inside-beedriver.1139905/
2022-04-27 20:02:18 -05:00
Frostebite 8abce48a48
Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt

* cloud-runner-system prefix changed to cloud-runner

* Update cloud-runner-aws-pipeline.yml

* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default

* github ephemeral pipeline support

* github ephemeral pipeline support

* Merge remote-tracking branch 'origin/main' into cloud-runner-develop

# Conflicts:
#	dist/index.js.map
#	src/model/cloud-runner/providers/aws/aws-task-runner.ts
#	src/model/cloud-runner/providers/aws/index.ts

* garbage collection

* garbage collection

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* self hosted runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* ephemeral runner pipeline

* download runner each time

* download runner each time

* download runner each time

* garbage collect all older than 1d as part of cleanup

* download runner each time

* number container cpu and memory for aws

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* Skip printing size unless cloudRunnerIntegrationTests is true

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* transition zip usage in cache to uncompressed tar for speed

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* per provider container defaults

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* baked in cloud formation template

* better aws commands

* better aws commands

* parse number for cloud formation template

* remove container resource defaults from actions yaml

* remove container resource defaults from actions yaml

* skip all input readers when cloud runner is local

* prefer fs/promises

* actually set aws cloud runner step as failure if unity build fails

* default to 3gb of ram - webgl fails on 2
2022-04-22 00:47:45 +01:00
Webber Takken 5ae03dfef6
Streamline code styles (#384)
* feat: streamline code styles

* feat: spacing for comments and return statements

* chore: enforce camelcase

* fix: remove npm lock file

* fix: add integrity test

* fix: remove logfile

* chore: update node in test workflow
2022-04-12 00:43:41 +02:00
gnomesley 4be5d2ddf4
Add username and password to mac build.sh (#383) 2022-04-11 22:22:52 +02:00
Frostebite a61c02481f
Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml

* Update cloud-runner-k8s-pipeline.yml

* yarn build

* yarn build

* correct branch ref

* correct branch ref passed to target repo

* Create k8s-tests.yml

* Delete k8s-tests.yml

* correct branch ref passed to target repo

* correct branch ref passed to target repo

* Always describe AWS tasks for now, because unstable error handling

* Remove unused tree commands

* Use lfs guid sum

* Simple override cache push

* Simple override cache push and pull override to allow pure cloud storage driven caching

* Removal of early branch (breaks lfs caching)

* Remove unused tree commands

* Update action.yml

* Update action.yml

* Support cache and input override commands as input + full support custom hooks

* Increase k8s timeout

* replace filename being appended for unknclear reason

* cache key should not contain whitespaces

* Always try and deploy rook for k8s

* Apply k8s files for rook

* Update action.yml

* Apply k8s files for rook

* Apply k8s files for rook

* cache test and action description for kuber storage class

* Correct test and implement dependency health check and start

* GCP-secret run, cache key

* lfs smudge set explicit and undo explicit

* Run using external secret provider to speed up input

* Update cloud-runner-aws-pipeline.yml

* Add nodejs as build step dependency

* Add nodejs as build step dependency

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Cloud Runner Tests must be specified to capture logs from cloud runner for tests

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs

* better defaults for new inputs

* better defaults

* merge latest

* force build update

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* use npm n to update node in unity builder

* correct new line

* quiet zipping

* quiet zipping

* default secrets for unity username and password

* default secrets for unity username and password

* ls active directory before lfs install

* Get cloud runner secrets from

* Get cloud runner secrets from

* Cleanup setup of default secrets

* Various fixes

* Cleanup setup of default secrets

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* Various fixes

* AWS secrets manager support

* less caching logs

* default k8s storage class to pd-standard

* more readable build commands

* Capture aws exit code 1 reliably

* Always replace /head from branch

* k8s default storage class to standard-rwo

* cleanup

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* further cleanup input

* folder sizes to inspect caching

* dir command for local cloud runner test

* k8s wait for pending because pvc will not create earlier

* prefer k8s standard storage

* handle empty string as cloud runner cluster input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* local-system is now used for cloud runner test implementation AND correctly unset test CLI input

* fix unterminated quote

* fix unterminated quote

* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name

* remove head and heads from branch prefix

* fix reversed caching direction of cache-push

* fixes

* fixes

* fixes

* cachePull cli

* fixes

* fixes

* fixes

* fixes

* fixes

* order cache test to be first

* order cache test to be first

* fixes

* populate cache key instead of using branch

* cleanup cli

* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files

* import cli methods

* import cli files explicitly

* import cli files explicitly

* import cli files explicitly

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* import cli methods

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* log parameters in cloud runner parameter test

* Cloud runner param test before caching because we have a fast local cache test now

* Using custom build path relative to repo root rather than project root

* aws-garbage-collect at end of pipeline

* aws-garbage-collect do not actually delete anything for now - just list

* remove some legacy du commands

* Update cloud-runner-aws-pipeline.yml

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* log contents after cache pull and fix some scenarios with duplicate secrets

* PR comments

* Replace guid with uuid package

* use fileExists lambda instead of stat to check file exists in caching

* build failed results in core error message

* Delete sample.txt
2022-04-11 00:00:37 +01:00