My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...

Running with gitlab-runner 13.4.1 (REDACTED) on runner-gitlab-runner-REDACTED-REDACTED REDACTED
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ...
Preparing environment
00:03
Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending
Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/
Created fresh repository.
Checking out REDACTED as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for REDACTEDREDACTEDREDACTEDREDACTED...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
00:05
$ git config --global credential.helper cache
$ git fetch
From * [new branch] dev -> origin/dev
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
$ echo ${TF_ACCOUNT}
REDACTED
$ echo ${TF_ADDRESS}
$ echo TF_HTTP_ADDRESS=${TF_ADDRESS}
TF_HTTP_ADDRESS=
$ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
TF_HTTP_LOCK_ADDRESS=
$ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
TF_HTTP_UNLOCK_ADDRESS=
$ echo TF_ADDRESS=${TF_ADDRESS}
TF_ADDRESS=
$ export TF_HTTP_ADDRESS=${TF_ADDRESS}
$ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
$ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
$ export TF_HTTP_LOCK_METHOD="POST"
$ export TF_HTTP_UNLOCK_METHOD="DELETE"
$ export TF_HTTP_RETRY_WAIT_MIN='5'
$ export TF_HTTP_USERNAME='JOHN.DOE'
$ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
$ export TF_VAR_ACCOUNT=${TF_ACCOUNT}
$ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
$ export AWS_DEFAULT_REGION='us-east-1'
$ terraform init
Initializing modules...
Downloading git:: for lambda...
- lambda in .terraform/modules/lambda
Downloading git:: for lambda_iam...
- lambda_iam in .terraform/modules/lambda_iam
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint requires auth
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

A peek at my gitlab-ci.yml:

image: name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 entrypoint: - '/usr/bin/env' - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
stages: - build - plan - apply
variables: PLAN: plan.tfplan JSON_PLAN_FILE: tfplan.json WORKSPACE: "dev" TF_IN_AUTOMATION: "true" ZIP_FILE: "lambda_package.zip" STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH} TF_ACCOUNT: "" ORG_ACCOUNT: "" TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH} TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
cache: key: "$CI_COMMIT_SHA" paths: - .terraform
before_script: - git config --global credential.helper cache - git fetch - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi - echo ${TF_ACCOUNT} - echo ${TF_ADDRESS} - echo TF_HTTP_ADDRESS=${TF_ADDRESS} - echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK} - echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} - echo TF_ADDRESS=${TF_ADDRESS} - export TF_HTTP_ADDRESS=${TF_ADDRESS} - export TF_HTTP_LOCK_ADDRESS=${TF_LOCK} - export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} - export TF_HTTP_LOCK_METHOD="POST" - export TF_HTTP_UNLOCK_METHOD="DELETE" - export TF_HTTP_RETRY_WAIT_MIN='5' - export TF_HTTP_USERNAME='MATTHEW.FETHEROLF' - export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring} - export TF_VAR_ACCOUNT=${TF_ACCOUNT} - export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT} - export AWS_DEFAULT_REGION='us-east-1' - terraform init
# -----BUILD----- # 1 build job per lambda function
buildLambda: stage: build tags: - cluster script: - echo "Beginning Build" - cd lambda_code/ - echo "Switched into Lambda dir" - pip3 install -r requirements.txt --target . - echo "Requirements installed" - echo $ZIP_FILE - zip -r $ZIP_FILE * - echo "Zip file packaged up" artifacts: paths: - lambda_code/ only: - dev - master - merge_requests
# -----PLAN-----
planMerge: stage: plan tags: - cluster script: - terraform plan dependencies: - buildLambda only: - merge_requests
planCommit: stage: plan tags: - cluster script: - terraform plan dependencies: - buildLambda
# -----APPLY----- # dev branch will auto deploy
applyDev: stage: apply tags: - cluster script: - echo "Running Terraform Apply" - terraform apply -auto-approve dependencies: - buildLambda only: - dev #when: manual # prod branch will require manual deployment approval
applyProd: stage: apply tags: - cluster script: - echo "Running Terraform Apply" - terraform apply -auto-approve when: manual dependencies: - buildLambda only: - master

backend.tf:

terraform { backend "http" { }
}

main.tf:

locals { common_tags = { SVC_ACCOUNT_ID = "REDACTED", CLOUD_PLATFORM_PROD_ID = "REDACTED" }
}
module "lambda" { source = "git::" lambda_name = var.name lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}" lambda_handler = var.handler lambda_runtime = var.runtime default_lambda_timeout = var.timeout env = local.common_tags ACCOUNT = var.ACCOUNT
}
module "lambda_iam" { source = "git::" lambda_policy = var.lambda_policy ACCOUNT = var.ACCOUNT lambda_role = var.lambda_role
}

inputs.tf:

variable "handler" { type = string default = "handler.lambda_handler"
}
variable "runtime" { type = string default = "python3.8"
}
variable "name" { type = string default = "onboard-cloud-account-into-monitoring"
}
variable "timeout"{ type = string default = "120"
}
variable "lambda_role" { type = string default = "cloud-platform-onboard-to-monitoring"
}
variable "ACCOUNT" { type = string
}
variable "ORG_ACCOUNT" { type = string
}
variable "lambda_policy" { default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}"
}

provider.tf:

provider "aws" { region = "us-east-1" assume_role { role_arn ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring" } default_tags { tags = { owner = "[email protected]" altowner = "[email protected]" blc = "REDACTED" costcenter = "REDACTED" itemid = "PlatformAutomation" } }
}

Perhaps I also need to share the terraform modules being invoked?

3

2 Answers

We found the problem: The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.

in my case there was no any protected environment variables but it was required to rename TF_STATE_NAME

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.