Resource should exist before deployment. Cannot read property 'getApplicationURL' of undefined.

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name:

AzureRmWebAppDeploymentV4 (AzureRMWebAppDeployment@4)

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises?

If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
https://dev.azure.com/Wych/Platform/_build/results?buildId=1730&view=logs&j=d1f3ad78-87a3-52d4-781d-88cb006f9122&t=3ca6baf2-e6c7-575f-3171-0693595489c7

Issue Description
Azure Web App Deployment can't find existing Microsoft.Web/sites
##[debug]Deployment Failed with Error: Error: Resource '...xxxx...' doesn't exist. Resource should exist before deployment.
This appears to be because the query is case sensitive:
##[debug][GET]https://management.azure.com/subscriptions/a1dc7c64-6f29-4584-90a3-a65732d701af/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01
Microsoft.Web%252FSites >Microsoft.Web/Sites but the correct value (and the value that is in other deployment strings is Microsoft.Web/sites note the lower case s in sites.
[Include task name(s), screenshots and any other relevant details]
Task logs
##[debug]{"subscriptionID":"","subscriptionName":"","servicePrincipalClientID":"","environmentAuthorityUrl":"https://login.windows.net/","tenantID":"","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"ServicePrincipal","activeDirectoryResourceID":"https://management.azure.com/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","scopeLevel":"Subscription","authenticationType":"","servicePrincipalKey":,"isADFSEnabled":false,"applicationTokenCredentials":{"clientId":"","domain":"","baseUrl":"https://management.azure.com/","authorityUrl":"https://login.windows.net/","activeDirectoryResourceId":"https://management.azure.com/","isAzureStackEnvironment":false,"authType":"","secret":,"isADFSEnabled":false}}
Got service connection details for Azure App Service:''
##[debug][POST]https://login.windows.net//oauth2/token/
##[debug][GET]https://management.azure.com/subscriptions//resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ ''&api-version=2016-07-01
##[debug]Correlation ID from ARM api call response : b79bdf30-6fd1-4ff5-97af-955641ac01b3
##[debug]Deployment Failed with Error: Error: Resource '' doesn't exist. Resource should exist before deployment.
(node:1722) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined
at ContainerWebAppDeploymentProvider. (/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.184.4/deploymentProvider/AzureRmWebAppDeploymentProvider.js:54:73)
##[debug]task result: Failed
at Generator.next ()
##[error]Error: Resource '' doesn't exist. Resource should exist before deployment.
##[debug]Processed: ##vso[task.issue type=error;]Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
[Insert error from the logs here for a quick overview]

Hello tomred-net
Can you please confirm if the web site / app is already created ? The task expects the app should be pre-created.
Thanks

Hi, @v-ibshaik
The App Service Plan and App Services exist and can be seen when querying the az resource list. The query name matches the search name.
This is the cleaned up output:
==============================================================================
Task         : Azure App Service deploy
Description  : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
Version      : 4.184.4
Author       : Microsoft Corporation
Help         : https://aka.ms/azureappservicetroubleshooting
==============================================================================
Got service connection details for Azure App Service:'request-to-pay-service'
##[error]Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.
(node:1689) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined

You will see the request that is failing is the filter resources by name and type. It is either the double escaping of the / in the Microsoft.Web%252FSites or if the query is case sensitive the use of S instead of s in sites as you can see the name matches the following result.
##[debug][GET]https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01
##[debug]Correlation ID from ARM api call response : b79bdf30-6fd1-4ff5-97af-955641ac01b3
##[debug]Deployment Failed with Error: Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.

    "id": "/subscriptions/.../resourceGroups/rs-core-production-australiaeast/providers/Microsoft.Web/serverFarms/request-to-pay-service-plan",
    "name": "request-to-pay-service-plan",
    "id": "/subscriptions/.../resourceGroups/rs-core-production-australiaeast/providers/Microsoft.Web/sites/request-to-pay-service",
    "name": "request-to-pay-service",

Hi @tomred-net
Could you please share the details like how you are creating the resources. Is it created via pipelines before deployment or already exists.
Thank you.

Hi @v-saikumart,
The App Service and App Service Plan resources are created using Terraform and have been provisioned for that last week.
The deployment step for the container is as follows:
variables:
  serviceConnection: pipeline-to-azure
  ContainerNamespace: acrproductionaustraliaeast.azurecr.io
  ContainerRepository: AcrProductionAustraliaeast
  WebAppName: wych-pay-services-webapp
  ContainerTag: latest

stages:
- stage: Deploy_App_Service
  - deployment: Deploy_App_Service
    strategy:
      runOnce: #rolling, canary are the other strategies that are supported
        deploy:
          steps:
          - checkout: self
            submodules: true
          - task: AzureRMWebAppDeployment@4
            displayName: Azure App Service Deploy
            inputs:
              appType: webAppContainer
              ConnectedServiceName: $(serviceConnection) # pipelines ARM serviceConnection
              WebAppName: $(WebAppName)                      # request-to-pay-service
              DockerNamespace: $(ContainerNamespace)     # xxx.azurecr.io
              DockerRepository: $(ContainerRepository)        # request-to-pay-service
              DockerImageTag: $(ContainerTag)                     # latest

@tomred-net
Can you share the details that you are experiencing the  issue consistently or just immediately after creating app from terraform. And can you please run the mentioned GET API call from postman and confirm on same.
Thank you.

HI @v-saikumart,
I am experiencing this consistently.
The query from the request fails:
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.WebWeb%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01

{
    "value": []
}

When I fix the URL encoding it passes
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01

{
    "value": [
        {
            "id": "/.../request-to-pay-service",
            "name": "request-to-pay-service",
            "type": "Microsoft.Web/sites",
            "kind": "app,linux,container",
            "location": "australiaeast",
            "identity": {
                "principalId": "...",
                "tenantId": "...",
                "type": "SystemAssigned"
            },
            "tags": {}
        }
    ]
}

Hi @tomred-net
Could you please try
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01 instead of
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01 and confirm on same.
Thank you.

@v-saikumart as demonstrated in my previous reply I tried the first url (containing  'Microsoft.Web%252FSites' ) which is the same as the value submitted by the azure task and the result is an empty array.
{
    "value": []
}

If I replace the double url encoding of '/' then I get the successful results as shown in my previous answer.
To clarify / is encoded to %2f which appears to be re-encoded to %252f as original % is encoded as %25.
url_encode('/') = %2f
url_encode( url_encode('/') ) = %252f

@v-saikumart @v-ibshaik is there anything else I can do to help you debug this issue?

@tomred-net  We are still investigating on this. Will update the status accordingly.
Thank you.

@tomred-net
Were you able to find a fix for this?  I am experiencing a similar issue, but it is sporadic.

Provision app service with Terraform
Wait 5 minutes (previously I had a 60 second wait and it was working fine for weeks).
Deploy web site via AzureWebApp@1 task

Up until yesterday this was working and now sometimes it works and other times it fails.
Error:
Got service connection details for Azure App Service:'app-xxxx-f48362-eaus'
##[error]Error: Resource 'app-xxxx-f48362-eaus' doesn't exist. Resource should exist before deployment.

@tomred-net
Were you able to find a fix for this? I am experiencing a similar issue, but it is sporadic.

Provision app service with Terraform
Wait 5 minutes (previously I had a 60 second wait and it was working fine for weeks).
Deploy web site via AzureWebApp@1 task

Up until yesterday this was working and now sometimes it works and other times it fails.
Error:
Got service connection details for Azure App Service:'app-xxxx-f48362-eaus'
##[error]Error: Resource 'app-xxxx-f48362-eaus' doesn't exist. Resource should exist before deployment.

I am also seeing the exact message. Was working up until today

Just started working today! I hope it's a permanent fix...

@tomred-net Could you please share the below mentioned details, as we are try to reproduce the issue and it is working as expected.
i. Complete Pipeline logs. ii. Complete YAML file.
Thank you.

Hi @v-saikumart,
I am sorry I did not receive a notification of you edits to your original reply, hence I did not notice the request for more information. I will run the deployment now and extract the logs. It is still failing for the same reason.
Dermot

@v-saikumart
Please find attached both the log file and the azure-pipeline.yml
Dermot
azure-pipeline.yml.txt
failed.deployment.txt

@RSteeno No fix so far. It is still happening for us.

@tomred-net  Thank you for the response. We will investigate further and share the status accordingly.
Thank you.

Same problem for me and my team of 15 devs, this is heavily impacting us currently. The only difference to the behavior reported by @tomred-net is that is does not fail always, but very often. This is even more annoying.

Same problem for me and my team of 15 devs, this is heavily impacting us currently. The only difference to the behavior reported by @tomred-net is that is does not fail always, but very often. This is even more annoying.

I have an case open with MSFT.  The engineer referenced an ongoing Resource Manager issue dated Jan 13th which could be related.
https://status.azure.com/en-us/status/history/

@tomred-net  Could please verify the SPN role and confirm the web app(request-to-pay-service) is present in that subscription.

I too am dealing with this issue and have a support case open with MS. Please let me know if there is anything I can provide to assist in resolving this issue as it is causing a lot of issues for our devs.

@tomred-net Could please verify the SPN role and confirm the web app(request-to-pay-service) is present in that subscription(PAYGO Subscription). Thank you.

Hi @v-saikumart, the request-to-pay-service is in the PAYGO Subscription. The Service Principle has Azure Service Management API permissions and is a 'Cloud application administrator'. The SPN is capable of deploying Static Web Apps to Microsoft.Web%2FStaticSites and Containers to Microsoft.ContainerRegistry%2Fregistries without issue.

Also dealing with this issue. Seems completely random if it fails or not. Deployed 4 apps just now. 3 of there were successful, but the last one refuses to deploy.

Having the same issue.
see https://developercommunity.visualstudio.com/t/Error:-Resource-DiagnosisCodeApi-doesn/1646720?viewtype=all&space=21&ftype=problem&sort=newest

@v-saikumart as demonstrated in my previous reply I tried the first url (containing 'Microsoft.Web%252FSites' ) which is the same as the value submitted by the azure task and the result is an empty array.
{
    "value": []
}

If I replace the double url encoding of '/' then I get the successful results as shown in my previous answer.
To clarify / is encoded to %2f which appears to be re-encoded to %252f as original % is encoded as %25. url_encode('/') = %2f url_encode( url_encode('/') ) = %252f

I have old resources working fine and resources cloned from an existing app service also working. Issue is just w/ newly created resources. However, both have the same encoding.
test-02 is cloned and working

[GET]https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxx/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'app-xxxxxxx-testing-02'&api-version=2016-07-01

testing01 is new

[GET]https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxxx/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'app-xxxxxxxxx-testing01'&api-version=2016-07-01

I am experiencing this same error with new Azure Functions. 1 out of 4 of them showed up in the drop down for the existing service connection and even a new service connection.

I can confirm the same problem on a newly created resource, it fails very very often

My team have been facing the same issue. It fails sporadically.

All of my Azure Functions have now showed up this morning that were created yesterday. They were not available as of 10:30-11pm last night after being created 4-5 hours previously.

We also have this issue now. Out of the 5 deploys I just did to the same Azure Function, only 3 were successfull.
2 of them failed with:
##[error]Error: Resource 'demoFunctionApppriv2' doesn't exist. Resource should exist before deployment.

The new resource showed up this morning, wasn't showing up yesterday.

@tomred-net  Is still issue exist, please confirm.
Thank you.

My new resource created on the 25th showed up too, however anything created today isn't showing up.
Clones aren't either showing up, this was working yesterday.
Please, please get this fixed ASAP. We're seriously impeded by this issue.

We are facing similar issue as well.
Basically what happens behind the curtains is that deployment tasks are calling the https://management.azure.com/subscriptions/{subscription-id}/resources API to validate the target resources of the deployments. Nowadays the result of this call does not contain every resource for some reason, if the target resource is missing, the deployment fails.

I have also recently experienced that commands such as "az resource list" not always gives correct results. E.g. a cosmos db account would should up in the Azure Portal and with "az cosmosdb list", but not with "az resource list".
Not a clue if there is a connection, but I figured it might be interesting.

Following yesterday deployment the pipeline seems to work fine today : does anyone experiencing only in new resources? Also does it occurs on  resources that are deployed from an ARM template? It seems to me that  the resource manager api has some consistency issue

This is really kind of bizarre.  I'm having the same issue, but it seem to only happen when I use an ARM template.
I was able to work around the issue initially by creating the resource (function app) through the portal.  I was then able to deploy to it fine from Azure Devops.
However, after deploying the ARM template (incremental mode), it stopped working right after.  I was hoping to still use the ARM template so that I wouldn't have to manually configure everything else about the app.

I'm seeing same issue as everyone above. My pipeline uses a bicep/arm template to deploy new app services and then deploy my application code to these newly created app services. This worked flawlessly until a couple weeks ago. Now I get ##[error]Error: Resource '{MyAppServiceName}' doesn't exist. Resource should exist before deployment. just about every time I run the deploy pipeline. I found if I wait 10-15 minutes after the deploy fails and "Rerun Failed Jobs" the second attempt will complete successfully. Unfortunately, I HAVE to wait that 10-15 minutes, so adding some custom retry logic or using the "Automatic retries for task" with "retryCountOnTaskFailure" wont work for me. This is really becoming a headache. Hopefully we get some help on this soon!

I have this issue aswell, when creating a WebApp through the Portal GUI. I'm able to select my WebApp through the release pipeline task when selecting the deployment resource. Resources has existed for more than an hour.

Same issue for me. Tried multiple things but none works out. I've created a duplicated ticket which is now closed: #15806. Hope this issue gets fixed soon because it's blocking our development flow.

I'm running into the same issue. My deployment process has been working without issue until a couple of weeks ago. Now it regularly fails with this error 'Resource should exist before deployment'.

I'm also running into the 'Resource should exist before deployment' error for 2 different tenants and subscriptions, both my personal, and for work.
Are there any workarounds to neatly rerun the failed job from the deployment? This is causing issues for our deployments.

FYI
I didn't check again recently, but my initial issue was with resources created manually in the GUI.

Hello to all,
since the deployment (by template or portal) does not seems to matter, could it be a regional issue? all the resources affected were for us  in West Europe

Hello to all, since the deployment (by template or portal) does not seems to matter, could it be a regional issue? all the resources affected were for us in West Europe

I just deployed a WebApp in West Europe en was facing the issues as described here (9 out of 10 deployments failed). Deploying the same app to Central US and no issue at all.

@adalfa all my deployments are in the West US 2 region.

@here We investigating on the issue, we will provide status on same at earliest.
Thank you

@v-saikumart If you need more information for understanding this issue please reach out.

@tomred-net We are unable to repro the issue. Is still issue exist, please confirm.
Thank you.

@v-saikumart yes - retried today and the issue continues to fail as before.

Hear are my diagnostic logs from a failure. All GUIDs obfuscated.
The exact same deployment with no changes ran successfully a minute before. I've had this problem with function apps, static web apps and app services... across multiple regions... for months.
YAML for the failing step:

task: AzureFunctionApp@1
inputs:
azureSubscription: 'Pay-As-You-Go ()'
appType: functionApp
appName: gvi-af-marketingmbreader
package: '$(Build.ArtifactStagingDirectory)//.zip'
displayName: 'Deploy Azure Function (Prod)'

Logs:
2022-02-02T20:26:08.1592842Z ##[debug]Evaluating condition for step: 'Deploy Azure Function (Dev)'
2022-02-02T20:26:08.1594305Z ##[debug]Evaluating: SucceededNode()
2022-02-02T20:26:08.1594820Z ##[debug]Evaluating SucceededNode:
2022-02-02T20:26:08.1595763Z ##[debug]=> True
2022-02-02T20:26:08.1596270Z ##[debug]Result: True
2022-02-02T20:26:08.1596804Z ##[section]Starting: Deploy Azure Function (Dev)
2022-02-02T20:26:08.1606420Z ==============================================================================
2022-02-02T20:26:08.1606752Z Task         : Azure Functions
2022-02-02T20:26:08.1607078Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2022-02-02T20:26:08.1607415Z Version      : 1.198.1
2022-02-02T20:26:08.1607614Z Author       : Microsoft Corporation
2022-02-02T20:26:08.1607882Z Help         : https://aka.ms/azurefunctiontroubleshooting
2022-02-02T20:26:08.1608197Z ==============================================================================
2022-02-02T20:26:08.1813451Z ##[debug]Using node path: /home/vsts/agents/2.198.2/externals/node10/bin/node
2022-02-02T20:26:08.2835265Z ##[debug]agent.TempDirectory=/home/vsts/work/temp
2022-02-02T20:26:08.2857925Z ##[debug]loading inputs and endpoints
2022-02-02T20:26:08.2867103Z ##[debug]loading INPUT_AZURESUBSCRIPTION
2022-02-02T20:26:08.2880452Z ##[debug]loading INPUT_APPTYPE
2022-02-02T20:26:08.2883267Z ##[debug]loading INPUT_APPNAME
2022-02-02T20:26:08.2885389Z ##[debug]loading INPUT_DEPLOYTOSLOTORASE
2022-02-02T20:26:08.2887298Z ##[debug]loading INPUT_SLOTNAME
2022-02-02T20:26:08.2889615Z ##[debug]loading INPUT_PACKAGE
2022-02-02T20:26:08.2891280Z ##[debug]loading INPUT_DEPLOYMENTMETHOD
2022-02-02T20:26:08.2893682Z ##[debug]loading ENDPOINT_AUTH***
2022-02-02T20:26:08.2900968Z ##[debug]loading ENDPOINT_AUTH_SCHEME_***
2022-02-02T20:26:08.2901891Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_TENANTID
2022-02-02T20:26:08.2902778Z ##[debug]loading ENDPOINT_AUTH_PARAMETERSERVICEPRINCIPALID
2022-02-02T20:26:08.2905467Z ##[debug]loading ENDPOINT_AUTH_PARAMETERAUTHENTICATIONTYPE
2022-02-02T20:26:08.2906383Z ##[debug]loading ENDPOINT_AUTH_PARAMETERSERVICEPRINCIPALKEY
2022-02-02T20:26:08.2907555Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2022-02-02T20:26:08.2908211Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2022-02-02T20:26:08.2910946Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2022-02-02T20:26:08.2911623Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2022-02-02T20:26:08.2912184Z ##[debug]loaded 17
2022-02-02T20:26:08.2912761Z ##[debug]Agent.ProxyUrl=undefined
2022-02-02T20:26:08.2913326Z ##[debug]Agent.CAInfo=undefined
2022-02-02T20:26:08.2913896Z ##[debug]Agent.ClientCert=undefined
2022-02-02T20:26:08.2914509Z ##[debug]Agent.SkipCertValidation=undefined
2022-02-02T20:26:08.3937264Z ##[debug]agent.proxyurl=undefined
2022-02-02T20:26:08.3938530Z ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
2022-02-02T20:26:08.3940072Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_build_1054_0
2022-02-02T20:26:08.4273737Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp
2022-02-02T20:26:08.5902454Z ##[debug]check path : /home/vsts/work/tasks/AzureFunctionApp/1.198.1/task.json
2022-02-02T20:26:08.5904548Z ##[debug]adding resource file: /home/vsts/work/tasks/AzureFunctionApp/1.198.1/task.json
2022-02-02T20:26:08.5905550Z ##[debug]system.culture=en-US
2022-02-02T20:26:08.5927343Z ##[debug]check path : /home/vsts/work/tasks/AzureFunctionApp/1.198.1/node_modules/azure-pipelines-tasks-azurermdeploycommon/module.json
2022-02-02T20:26:08.5928781Z ##[debug]adding resource file: /home/vsts/work/tasks/AzureFunctionApp/1.198.1/node_modules/azure-pipelines-tasks-azurermdeploycommon/module.json
2022-02-02T20:26:08.5930255Z ##[debug]system.culture=en-US
2022-02-02T20:26:08.5949280Z ##[debug]azureSubscription=
2022-02-02T20:26:08.5950029Z ##[debug]appType=functionApp
2022-02-02T20:26:08.5950771Z ##[debug]deployToSlotOrASE=false
2022-02-02T20:26:08.5951360Z ##[debug]customWebConfig=undefined
2022-02-02T20:26:08.5951929Z ##[debug]appSettings=undefined
2022-02-02T20:26:08.5954540Z ##[debug]startUpCommand=undefined
2022-02-02T20:26:08.5955157Z ##[debug]configurationStrings=undefined
2022-02-02T20:26:08.5962148Z ##[debug]appName=gvi-af-marketingmbreader
2022-02-02T20:26:08.5962760Z ##[debug]SlotName : production
2022-02-02T20:26:08.5963528Z ##[debug]*** auth scheme = ServicePrincipal
2022-02-02T20:26:08.5965406Z ##[debug]*** data subscriptionid = ***
2022-02-02T20:26:08.5966300Z ##[debug]*** data subscriptionname = Pay-As-You-Go
2022-02-02T20:26:08.5967699Z ##[debug]*** auth param serviceprincipalid = ***
2022-02-02T20:26:08.5968583Z ##[debug]*** data environmentAuthorityUrl = https://login.windows.net/
2022-02-02T20:26:08.5969503Z ##[debug]*** auth param tenantid = ***
2022-02-02T20:26:08.5970379Z ##[debug]=https://management.azure.com/
2022-02-02T20:26:08.5971182Z ##[debug] data environment = AzureCloud
2022-02-02T20:26:08.5971996Z ##[debug]*** auth scheme = ServicePrincipal
2022-02-02T20:26:08.5973307Z ##[debug]*** data msiclientId = undefined
2022-02-02T20:26:08.5974309Z ##[debug]*** data activeDirectoryServiceEndpointResourceId = https://management.core.windows.net/
2022-02-02T20:26:08.5975290Z ##[debug]*** data AzureKeyVaultServiceEndpointResourceId = https://vault.azure.net
2022-02-02T20:26:08.5976207Z ##[debug]*** data AzureKeyVaultDnsSuffix = vault.azure.net
2022-02-02T20:26:08.5977274Z ##[debug]*** auth param authenticationType = spnKey
2022-02-02T20:26:08.5977942Z ##[debug]credentials spn endpoint
2022-02-02T20:26:08.5979394Z ##[debug]*** auth param serviceprincipalkey = ***
2022-02-02T20:26:08.5980255Z ##[debug]*** data EnableAdfsAuthentication = false
2022-02-02T20:26:08.5995735Z ##[debug]{"subscriptionID":"","subscriptionName":"Pay-As-You-Go","servicePrincipalClientID":"","environmentAuthorityUrl":"https://login.windows.net/","tenantID":"","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"ServicePrincipal","activeDirectoryResourceID":"https://management.azure.com/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","authenticationType":"spnKey","servicePrincipalKey":,"isADFSEnabled":false,"applicationTokenCredentials":{"clientId":"","domain":"","baseUrl":"https://management.azure.com/","authorityUrl":"https://login.windows.net/","activeDirectoryResourceId":"https://management.azure.com/","isAzureStackEnvironment":false,"authType":"spnKey","secret":,"isADFSEnabled":false}}
2022-02-02T20:26:08.5998359Z Got service connection details for Azure App Service:'gvi-af-marketingmbreader'
2022-02-02T20:26:08.6001082Z ##[debug][POST]https://login.windows.net//oauth2/token/
2022-02-02T20:26:08.7098638Z ##[debug][GET]https://management.azure.com/subscriptions/***/resources?$filter=resourceType EQ 'Microsoft.Web%2FSites' AND name EQ 'gvi-af-marketingmbreader'&api-version=2016-07-01
2022-02-02T20:26:08.9316165Z ##[debug]Correlation ID from ARM api call response : ***
2022-02-02T20:26:08.9317234Z ##[debug]Deployment Failed with Error: Error: Resource 'gvi-af-marketingmbreader' doesn't exist. Resource should exist before deployment.
2022-02-02T20:26:08.9318006Z ##[debug]task result: Failed
2022-02-02T20:26:08.9362219Z ##[error]Error: Resource 'gvi-af-marketingmbreader' doesn't exist. Resource should exist before deployment.
2022-02-02T20:26:08.9371634Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Resource 'gvi-af-marketingmbreader' doesn't exist. Resource should exist before deployment.
2022-02-02T20:26:08.9374551Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Resource 'gvi-af-marketingmbreader' doesn't exist. Resource should exist before deployment.
2022-02-02T20:26:08.9375405Z ##[debug]Deployment failed
2022-02-02T20:26:08.9378043Z ##[section]Finishing: Deploy Azure Function (Dev)

Today I did some additional testing but this time I've tried using different agents.  We have a pool of self-hosted windows 2016 agents in our environment, running the same exact pipeline on our agents has been successful 7 times today with 0 failures.
Here are my results using the MSFT Hosted Agents

Agent Type
Operating System
Agent Version
Results

MS Hosted
Ubuntu-latest
2.198.2
Fails

MS Hosted
ubuntu-18.04
2.198.2
Fails

MS Hosted
windows-latest
2.198.2
Fails

Self-hosted
Windows 2016
2.183.1
Success

Anyone else try a self-hosted agent?
I still have a case open with MSFT, but haven't gotten too far yet.
##[debug]Deployment Failed with Error: Error: Resource 'app-***-****-eaus' doesn't exist. Resource should exist before deployment.
##[debug]task result: Failed
##[error]Error: Resource 'app-***-***-eaus' doesn't exist. Resource should exist before deployment.

I was actually trying to get away from self-hosted agents, but now that you mention it, I have an older pipeline that uses a self-hosted agent, windows-2016 I think that still works. Maybe a workaround for now, but I too have an open case, which seems to be going nowhere. Alas, thanks for the tip on the self-hosted agent.

I run into very similar issue, in my pipeline I created a resource and then use AzureRmWebAppDeployment or AzureFunctionApp to deploy web apps or functions to newly created resources. In such case some times I get:

Got service connection details for Azure App Service: ''
##[error]Error: Resource '' doesn't exist. Resource should exist before deployment.
(node:1685) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined
at WindowsWebAppRunFromZipProvider. (/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.198.0/deploymentProvider/WindowsWebAppRunFromZipProvider.js:68:77)
at Generator.next ()
at fulfilled (/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.198.0/deploymentProvider/WindowsWebAppRunFromZipProvider.js:5:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:1685) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:1685) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Failed to add release annotation. TypeError: Cannot read property 'getApplicationSettings' of undefined

At this point of time resource exists in azure portal, but it seems AzureRmWebAppDeployment does not see it. If re run pipeline again with just deployment steps it succeed

Hi everyone, I explicitly added the resourceGroupName for the AzureWebAppContainer@1 task and it worked for me after:
steps:
  - task: AzureWebAppContainer@1
    displayName: 'Azure Web App on Container Deploy'
    inputs:
      azureSubscription: $(azureSubscription)
      appName: $(webAppName)
      deployToSlotOrASE: $(deployToSlotOrASE)
      slotName: $(slotName)
      containers: $(containers)
      resourceGroupName: $(resourceGroupName)

@ihales-mac I have already added the resourceGroupName to the task parameters and still getting the error.

@ihales-mac I have already added the resourceGroupName to the task parameters and still getting the error.

sorry for that @maikelvanhaaren, I have removed the comment, I think it worked by chance, second deployed failed again after that for me

@maikelvanhaaren so I'm reposting this again, small configuration mistake on my side after I set the resource group as a parameter but I confirm that explicitly setting the resource group works for me now:
steps:
  - task: AzureWebAppContainer@1
    displayName: 'Azure Web App on Container Deploy'
    inputs:
      azureSubscription: $(azureSubscription)
      appName: $(webAppName)
      deployToSlotOrASE: $(deployToSlotOrASE)
      slotName: $(slotName)
      containers: $(containers)
      resourceGroupName: $(resourceGroupName)

Note: this works for me when using the AzureWebAppContainer@1 type of deployment, might apply or not to other types of deployments

@v-saikumart Any update regarding this issue?

@maikelvanhaaren  We are working on it, will provide the update at earliest on same.
Thank you.

@maikelvanhaaren so I'm reposting this again, small configuration mistake on my side after I set the resource group as a parameter but I confirm that explicitly setting the resource group works for me now:
steps:
  - task: AzureWebAppContainer@1
    displayName: 'Azure Web App on Container Deploy'
    inputs:
      azureSubscription: $(azureSubscription)
      appName: $(webAppName)
      deployToSlotOrASE: $(deployToSlotOrASE)
      slotName: $(slotName)
      containers: $(containers)
      resourceGroupName: $(resourceGroupName)

Note: this works for me when using the AzureWebAppContainer@1 type of deployment, might apply or not to other types of deployments

Hi @ihales-mac we have tried with resource group but still we see the error. As a point of differentiation we are using
task: AzureRMWebAppDeployment@4. I will try task: AzureWebAppContainer@1 today.
D

@maikelvanhaaren so I'm reposting this again, small configuration mistake on my side after I set the resource group as a parameter but I confirm that explicitly setting the resource group works for me now:
steps:
  - task: AzureWebAppContainer@1
    displayName: 'Azure Web App on Container Deploy'
    inputs:
      azureSubscription: $(azureSubscription)
      appName: $(webAppName)
      deployToSlotOrASE: $(deployToSlotOrASE)
      slotName: $(slotName)
      containers: $(containers)
      resourceGroupName: $(resourceGroupName)

Note: this works for me when using the AzureWebAppContainer@1 type of deployment, might apply or not to other types of deployments

Hi @ihales-mac we have tried with resource group but still we see the error. As a point of differentiation we are using task: AzureRMWebAppDeployment@4. I will try task: AzureWebAppContainer@1 today.
D

First of all, I'm very thankfuly to see these answers. I'm having the same problem here and I'm be able to finish my pipeline successfully thanks yours hint.
But I have another question. How did you edit the steps in Release pipeline? I didn't find any option to edit a specific task in yaml format.
To do that I export the entire release pipeline in json format, edit the file, fill the resourceGroupName property and import the file back to devops.
Is there any easy way to fill that information?

Hi @ihales-mac we have tried with resource group but still we see the error. As a point of differentiation we are using
task: AzureRMWebAppDeployment@4. I will try task: AzureWebAppContainer@1 today.

@tomred-net,  Could you try it? Maybe the best solution is to change AzureRMWebAppDeployment by AzureWebAppContainer in this case.

Hi @angelots

Hi @ihales-mac we have tried with resource group but still we see the error. As a point of differentiation we are using
task: AzureRMWebAppDeployment@4. I will try task: AzureWebAppContainer@1 today.

@tomred-net, Could you try it? Maybe the best solution is to change AzureRMWebAppDeployment by AzureWebAppContainer in this case.

It didn't make a difference for me. Tried with AzureWebAppContainer@1 today.

Same issue here. After creating a resource in an existing resource group, the resource is invisible to release pipelines for 15 minutes or so. The service principal permissions are inherited from the resource group.
During these 15 minutes Azure DevOps release pipelines can see other older resources in the resource group, but not the newly created resource.
Region: East US.
I hope this helps.

@maikelvanhaaren  Could you please share Correlation ID.
Thank you.

@v-saikumart Is this what you requested?
Got service connection details for Azure App Service:'app-cms-____-PR-32543'
##[debug][POST]https://login.windows.net/____/oauth2/token/
##[debug][GET]https://management.azure.com/subscriptions/____/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'app-cms-____-PR-32543'&api-version=2016-07-01
##[debug]Correlation ID from ARM api call response : 322d7091-771d-45ac-8ebd-7274ee24e79f
##[debug]Deployment Failed with Error: Error: Resource 'app-cms-____-PR-32543' doesn't exist. Resource should exist before deployment.
##[debug]task result: Failed
##[error]Error: Resource 'app-cms-____-PR-32543' doesn't exist. Resource should exist before deployment.

This is happening even for West Europe. Tested for both Azure webapps for Linux and windows

Same issue here - please let us know if you need any additional information.
##[debug]Correlation ID from ARM api call response : acbc967c-0914-4129-befa-6e6417085f03
##[debug]Correlation ID from ARM api call response : 230036af-d450-44d7-bc08-83d8cab804cb

There seems to be an issue with the ARM's "resources" endpoint, which is hit by the "Azure App Service Deploy" task to find the resource (the app), while filtering for the resource type and name. The endpoint doesn't always return the resource.
Endpoint:
https://docs.microsoft.com/en-us/rest/api/resources/resources/list
Corresponding Azure CLI command:
https://docs.microsoft.com/en-us/cli/azure/resource?view=azure-cli-latest#az-resource-list
One solution is to use the "Azure Web App" task instead, while specifying the resource group name (resourceGroupName):
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-rm-web-app
With the resource group name specified, the task's behavior changes such that the "resources" endpoint is not hit.
Note that the "Azure App Service Deploy" task continues to hit the ARM endpoint, even with the resource group specified.

@v-saikumart Do you have any update on this issue?

Same issue here!

I'm having the same issue, any news?

Jumping in here. Our team has the same issue and it's incredibly frustrating.
We have an App Service that's serving 4x individual Function Apps. We use the AzureFunctionApp@1 task for deploying each of these Function Apps, and invariably at least one of the four fails. Retry after retry after retry, and eventually they start to succeed.
This significantly impacts our releases because we have several environments to deploy to as we make our way to prod, and now a developer is tied up clicking a retry button instead of doing real work.

even with the resource group specified.

Hi @FHajHusein, does "Azure Web App Deploy" work with App Service For Containers?

Hi Angelotas, why not using the "Azure Web App for Container" task?
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-rm-web-app-containers

@maikelvanhaaren We are working on it, will provide the update at earliest on same.
Thank you.
@v-saikumart, I have an active support case open that is with the Azure DevOps Support team.   I've asked the support engineer to reach out to you to collaborate on the issue.  I've included my case # for reference:<PHONE_NUMBER>008155

I have the same issue, it's really frustrating 😞

@maikelvanhaaren We are working on it, will provide the update at earliest on same.
Thank you.
@v-saikumart, I have an active support case open that is with the Azure DevOps Support team.   I've asked the support engineer to reach out to you to collaborate on the issue.  I've included my case # for reference:<PHONE_NUMBER>008155

@RSteeno ,
@v-saikumart  is not working on this issue now. Can you please inform the support engineer to reach out to me. Thanks
cc: @PhilipsonJoseph

@v-nagarajku We have an open ticket at the Azure App Services with number<PHONE_NUMBER>001295. We've shared the link to this ticket multiple times.

We are also having this issue and its a blocker for releases.

@maikelvanhaaren  we are not from the Azure App Services team. We are from azure-pipelines-task team. As  a luck or initial workaround can you please add a delay task before the task (e.g: Azure App deploy) and check.

Delay task won't solve this issue. Just for investigation, we've executed the https://management.azure.com/subscriptions/{subscriptionId}/resources call every minute for several hours to see if the Azure Mangement API returns our existing resources... there were missing things 5 hours after the successful resource provisioning.
The deployment task validates the existence of the target resources via executing the same Management API call.
This is not just a deployment or app/functionapp service group related issue, the management API returns invalid result hours after a successful deployment. Microsoft should take this more seriously.

@bmoore-msft raising for visibility

@borondy This is my experience as well. I was on a call with an ARM engineer earlier today. I'm going to point him to this thread to maybe get some visibility going here. I've had a support case open since this started happening on 1/12.

@safari137 @adalfa I can confirm that the problems occur when using ARM template deployments. I can't comment on the question whether it works if deployed through the Azure Portal. All of our deployments are for West Europe region, so we can rule out that it's specific to any location.
We are dynamically provisioning environments with around 12 different web apps, and typically ~4 of them are not deployable right after ARM deployment is complete. Overall, it's a rather complex ARM deployment with almost 9000 lines of Json - maybe that's an indicator as well. We also pinged the management endpoint manually before, during and after ARM deployment (as well as during and after web app deployment) and can confirm that for those resources that are not being found by the management API, the deployment jobs fail.
Adding a delay (or the relatively new retry functionality) does not help. It can take many minutes to many hours until those resources suddenly become visible through the API endpoint, without any further action like retries. It seems like for some resources, the endpoint takes a lot of time to become "eventually consistant".

@v-nagarajku As a shout out to the Azure DevOps team, I feel that this discussion is actually misplaced here, as in my opinion, the task implementation of Azure DevOps is not to blame. It's the underlying API that causes these problems.
Since we're in the repo for azure pipeline tasks, I want to share a solution (or rather a workaround) that we implemented in our team and that completely solved deployment hiccups we experienced. As a heads up: This particular solution only works when using Docker images for deployment.
TL;DR: Instead of using the built-in deployment task, use an app settings config change
Previous approach:
We used to use AzureRmWebAppDeployment@4 like this:
- task: AzureRmWebAppDeployment@4
  displayName: 'Docker Deploy '
  inputs:
    azureSubscription: ${{parameters.subscriptionName}}
    appType: webAppContainer
    WebAppName: $(resourceName)
    DockerNamespace: $(DOCKER_REGISTRY_NAMESPACE)
    DockerRepository: '${{parameters.dockerImageName}}:$(Build.BuildNumber)'

Problem: The implementation of AzureRmWebAppDeployment tries to make sure that the resource exists before deployment, so it calls out to the management API first, which causes the problems that brought us all here.
Our current solution:
We ditched AzureRmWebAppDeployment and use az webapp config appsettings set instead to only update the DOCKER_CUSTOM_IMAGE_NAME setting on our 12 target web apps with container support. (Note: All the other app settings have been written before by the ARM deployment as we handle infrastructure and configuration in one go. YMMV)
- task: AzureCLI@2
  displayName: 'Config Deploy'
  inputs:
    azureSubscription: ${{parameters.subscriptionName}}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az webapp config appsettings set --settings DOCKER_CUSTOM_IMAGE_NAME=$(DOCKER_REGISTRY_NAMESPACE)/${{parameters.dockerImageName}}:$(Build.BuildNumber) -g $(resourceGroup) -n $(resourceName)

This causes the container web app to start pulling the new image version that just got configured, create a container from it and spin it up, replacing the previous version. The effect is just like a regular deployment, but with one important difference: az webapp config appsettings set apparently does not care (and does not try to check) if the resource already exists.
This is already working for us for over 2 weeks with 0% error rate and with no apparent drawback (other than the fact that it does not show up as deployment in the web app portal).
Timing-wise, this call by itself is faster than the deployment task. However, we are waiting for the new version specifically to be up and running before continuing (details available if of interest), and then the time is pretty much exactly the same as for the regular deployment task.

Just want to mention my observation:
Resource in Azure Portal exists:

It does make NO difference which Task i use:
AzureWebApp@1 fails with: ##[error]Error: Resource 'XXXX-api-dev' doesn't exist. Resource should exist before deployment.
- task: AzureWebApp@1
    displayName: "Deploy Api"
    inputs:
        azureSubscription: $(azureSubscriptionDev)
        appName: "XXXX-api-dev"
        package: "$(System.ArtifactsDirectory)/**/*.zip"

AzureRmWebAppDeployment@4 fails with: ##[error]Error: Resource 'XXXX-api-dev' doesn't exist. Resource should exist before deployment.
- task: AzureRmWebAppDeployment@4
    displayName: "Deploy Api"
    inputs:
        ConnectionType: "AzureRM"
        azureSubscription: $(azureSubscriptionDev)
        appType: "webApp"
        WebAppName: "XXXX-api-dev"
        packageForLinux: "$(System.ArtifactsDirectory)/**/*.zip"

We have deployed over 60 apps in the past with AzureRmWebAppDeployment@4 with no issues.
I am facing this problem for 24h hours now. I was hoping it would randomly run successful. But no "luck".
I also DELETED my resource XXXX-api-dev in Azure Portal.
After recreation of the resource i redeployed with the same error ##[error]Error: Resource 'XXXX-api-dev' doesn't exist. Resource should exist before deployment.
Guys, this is getting really costly right now.

@enterprisebug There are currently two proposed solutions around that you can try:

@FHajHusein 's comment (direct link to it) that uses AzureWebApp task as you tried, but specifying the resource group as well, because this apparently omits the call to the erroneous API
Optionally, my solution also works if docker images are used for deployment, but this is not the case for you

@enterprisebug There are currently two proposed solutions around that you can try:

@FHajHusein 's comment (direct link to it) that uses AzureWebApp task as you tried, but specifying the resource group as well, because this apparently omits the call to the erroneous API
Optionally, my solution also works if docker images are used for deployment, but this is not the case for you

All solutions rely on using Azure DevOps tasks in a way that they don't call out to management.azure.com to check if a resource exists before deployment, because this API seems to have consistency issues. A support engineer from our open case with MSFT phrased the issue as follows:

When a new resource is created the PUT request for it gets routed through a certain Azure region based on factors like location, latency and so on. This region then becomes the "home" region of the resource metadata. For other regions to be aware of the existence of the new resource the metadata needs to be replicated across all regions. This means that when a GET request for a resource is routed through any other region than the "home" region of the resource, the resource may not be visible if the metadata replication hasn't finished. Usually this shouldn't take more than a few minutes. Lately some accounts have seen an increase in replication times caused mostly due to high load in the most common regions.

He also mentioned that the responsible product group is aware of the issue and working on it

I've also heard that this could be a replication issue for the ARM API.  Another workaround is to use a self-hosted agent which doesn't require any changes to your pipelines.  I have had 100% success when using a self-hosted agent.

Also experiencing this issue.  We're seeing delays of 5+ hours in API showing current state.

I finally abandoned this pipeline action in favor of the Azure CLI one.  It's actually a pretty straight forward replacement.  Here's the before/after YAML:

task: AzureWebApp@1
displayName: 'Azure Web Deploy'
inputs:
azureSubscription: 'sssss'
appType: webApp
appName: aaaaaaaaaa
package: '$(System.DefaultWorkingDirectory)/Web.zip'

task: AzureCLI@2
displayName: 'Azure CLI - Deploy Web'
inputs:
azureSubscription: 'sssss'
scriptType: ps
scriptLocation: inlineScript
inlineScript: 'az webapp deploy --resource-group rrrrrrrrr --name aaaaaaaaaa --src-path '$(System.DefaultWorkingDirectory)/Web.zip'

Just got a response from Azure Support saying the following:

This issue came up on a thread internally with the Team around 2/18 regarding a similar issue as multiple customers were reporting issues like “Resource ‘***’ doesn't exist. Resource should exist before deployment” while using Azure web app\Azure function app tasks.  And the Product Group is reporting that the issue should be mitigated and resolved now.

I'm about to start the process for a new release on our end, but seems like the problem is potentially fixed now by Azure. Will report back here soon.

Thanks for the update.  I have tested my deployment and it is still failing.  I'm going to ping the MSFT engineering working on my case to see if he's heard anything.

Just attempted on my end and the problem is definitely still there. Will respond to Microsoft support and attempt one of the workarounds mentioned in the thread, too.

Attempted doing the workaround mentioned in this comment.
Specifically, I updated the ask from AzureFunctionApp@1 (which we had been using but broke suddenly) to AzureWebApp@1. Turns out I was already specifying resourceGroupName, but I went ahead and gave it a shot. Unfortunately I encountered a different issue:

Error: Couldn't detect a version for the platform 'dotnet' in the repo.

Looks like this is an Oryx deployment issue that, laughably, has an issue that was closed with no resolution.
Instead of continuing down that rabbit hole, my next attempt is going to use the AzureCLI@2 task and attempt an az webapp deploy as suggested in this comment. From the documentation, it appears that az webapp deploy should work for deployment of FunctionApps as well as WebApps.

Attempted doing the az webapp deploy as suggested in this comment. Had to modify slightly because I'm running the task on a Linux Agent, so the I updated the script type to pscore.
Unfortunately, no dice there either. Failure was due to the Linux VM not actually supporting that command:

WARNING: This command is in preview and under ***elopment. Reference and support levels: https://aka.ms/CLI_refstatus
WARNING: Deployment type: static. To override deloyment type, please specify the --type parameter. Possible values: war, jar, ear, zip, startup, script, static
ERROR: This API isn't available in this environment yet!

🤷♂ 🤦♂
One of the two workarounds may be workable for Function App deploys given some additional investigation/testing, but no luck given the attempts I've made.

I finally abandoned this pipeline action in favor of the Azure CLI one. It's actually a pretty straight forward replacement. Here's the before/after YAML:

task: AzureWebApp@1
displayName: 'Azure Web Deploy'
inputs:
azureSubscription: 'sssss'
appType: webApp
appName: aaaaaaaaaa
package: '$(System.DefaultWorkingDirectory)/Web.zip'
task: AzureCLI@2
displayName: 'Azure CLI - Deploy Web'
inputs:
azureSubscription: 'sssss'
scriptType: ps
scriptLocation: inlineScript
inlineScript: 'az webapp deploy --resource-group rrrrrrrrr --name aaaaaaaaaa --src-path '$(System.DefaultWorkingDirectory)/Web.zip'

az webapp deploy doesn't seem to support the options available in the task, such as App Offline or deploy from zip etc. (can someone confirm this?) In that case, we might as well script everything out or rewrite the task.
Then we're basically losing the "ease of use" implied in using the Azure DevOps platform...

We are also having this issue and its a blocker for releases.
Edited: We restarted the app service in the portal and redid the pipeline and it worked, not sure if its related or not.

This also worked for me (restart, stopped and started again App Service)

We are also having this issue and its a blocker for releases.
Edited: We restarted the app service in the portal and redid the pipeline and it worked, not sure if its related or not.

This also worked for me (restart, stopped and started again App Service)

Not working for me.

It is a timing issue. Sometimes folks would try different things, and all of the sudden it starts working, so they think that is what fixed it. But what really ‘fixed’ it was the time it took to propagate.

@Sergey-Nosov-CloudFit , could you please explain on what you mean by the "timing issue"? I have the same problem, resource (Azure app service) exist, I could deploy to the app service from my visual studio fine but from the release pipeline in azure devops, it fails regularly ..nd sometimes is juts works. quite random

@Sergey-Nosov-CloudFit , could you please explain on what you mean by the "timing issue"? I have the same problem, resource (Azure app service) exist, I could deploy to the app service from my visual studio fine but from the release pipeline in azure devops, it fails regularly ..nd sometimes is juts works. quite random

Latency, propagation, delay... Basically you have to sit and wait until it starts working. It does not work immediately, and then it starts working after a while. HTH

Thank you..so basically I can't create a release pipeline and trust it to just work in devOps? I have to always sit and redploy in case it fails

If the same pipeline (or a different pipeline executed shortly before) creates the Azure resources targeted for deployment, then yes you are affected. Please read the earlier posts in this thread for possible workarounds.
If you are trying to deploy to Azure resources that are over 24 hours old, than most likely you are having a different issue.

I never had this bug until today, it's very frustrating that this is happening right now.

I see, indeed I tried to deploy to the app service shortly after creation. Hopefully something would be done about this..it is not okay. Thanks @Sergey-Nosov-CloudFit

Today I got the same problem as well. App service plan is on Linux and the application I wanted to deploy is for NodeJS. Last time it was enough to stop and start the application in the portal to make it work 😢

Started a deployment of 4 Function Apps (Linux function apps on consumption plan, all utilizing the same app service plan under the hood) before noon today. Getting close to end of business today and I'm stuck hitting "rerun failed jobs" over and over because one of the four fails to succeed.
This is pretty outrageous.

Welcome to the party. This has been happening for well over a month now. I think "outrageous" barely scratches the surface.

Are there more people in contact with Microsoft about this issue through an besides me? We have an Azure Support Plan.

Are there more people in contact with Microsoft about this issue through an besides me? We have an Azure Support Plan.

@maikelvanhaaren, yes, a few of us have tickets open with Microsoft on this issue.

Are there more people in contact with Microsoft about this issue through an besides me? We have an Azure Support Plan.

Looks like it was being addressed here too, https://developercommunity.visualstudio.com/t/Error:-Resource-DiagnosisCodeApi-doesn/1646720?viewtype=all&space=21&ftype=problem&sort=newest

My current workaround is to use a self-hosted agent.  Issue seems to be isolated to using the Microsoft-hosted agents.

My current workaround is to use a self-hosted agent. Issue seems to be isolated to using the Microsoft-hosted agents.

Unfortunately it is not. Happening on self-hosted agents as well for quite some time now (West Europe here).

Hello to all,
In my understanding with several cases opened we do not neither received any suggestions from Microsoft or any official acknowledgement of the issue. I understand that  this issue is not related on the pipeline's teams, but this remains the only place where some discussion has taken place and after a month is becoming really frustrating to have no definitive solution...

I don't know if it's a coincidence, but it seems that all error handling for pipeline yaml syntax is handled as no appservice.
For example, if "deployToSlotOrASE" is enabled in "AzureRmWebAppDeployment@4", there are parameters (ResourceGroupName, SlotName) that must be specified as required. If the required parameter is not specified, an error is treated as appservice does not exist.

@nadesu @v-nagarajku Any update?

New stuff here that seems related: https://status.azure.com/en-us/status/history/
I have also noticed some builds working on the first run today as well. Not 100%, but I'm starting to think that something is happening on the Azure end of things.

New stuff here that seems related: https://status.azure.com/en-us/status/history/
I have also noticed some builds working on the first run today as well. Not 100%, but I'm starting to think that something is happening on the Azure end of things.

We are still seeing the issue, so don't get your hopes up to high.

Any update on this issue? I'm facing the same problem with a manually created Web App.

Just received an e-mail from Microsoft where they finally acknowledge the issue:

Latest update:
The ARM team does not have a quick fix on this issue.
SOS Replication Delays
Root Cause Statement
Azure is a multi-regional product, where relative information of resources and their state needs to be replicated across multiple regions. Essentially, while the resource data itself resides in the selected region, metadata related to managing the resource, such as the resource id and its data location, is replicated across the various regions. This allows Azure to accept a resource management request from anywhere in the world and route the request to the closest region based on latency. This selection process is independent of the HTTP request body; even if the request has the notion of a 'location' associated with it. Through this model, we’re able to provide better performance for Azure’s management layer from any location while keeping the data securely stored in the configured region.
However, our investigation discovered that the issue you encountered was caused by a transient delay in the replication flow of resource metadata between Azure’s {source region} and {destination region} regions. We recognize that this is as a problem, and we are continuously working on improving the platform.
The Azure Resource Manager (ARM) team is actively iterating on significant improvements to the replication flows and mechanisms, including large-scale enhancements of our replication architecture. Meanwhile, we recommend that you use Azure Resource Graph whenever possible for resource list calls. Azure Resource Graph offers a one-minute SLO and is designed to performantly handle list calls at scale.
Work around:
So, we have to look for a work around. Since Azure App service deploy tasks checks the existence of the resource before deploying and this is where the issue is happening.
We have requested the App service team to provide us the az cli/powershell commands to deploy the packages to app services so that you can implement this workaround until we have a permanent fix from the ARM team.

I'm asking the support for my case if they had an estimate on the time needed for the consistency of the metadata, many thanks for reporting the finding, a very faint light at the end of the tunnel...

We have more than 50 pipelines that have trouble with this issue. We changed the deployment stage to custom az cli commands.

loluzapaluza, same problem, checked everyting and its not working at all...
##[error]Error: Resource 'tailspin-space-game-123456' doesn't exist. Resource should exist before deployment.

Hello to all, the issue still exists to my finding

Microsoft Support sent me a workaround for this issue. They told me to use the azure cli/powershell for deploying apps, functions and ARM templates:

Below are the useful links to carry out the deployment using
the cli:
•	Run your app in Azure App Service directly from a ZIP package: https://docs.microsoft.com/en-us/azure/app-service/deploy-run-package
•	Automate resource deployment for your function app in Azure Functions: https://docs.microsoft.com/en-us/azure/azure-functions/functions-infrastructure-as-code?tabs=windows
•	Deploy resources with ARM templates and Azure PowerShell: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-powershell
•	How to use Azure Resource Manager (ARM) deployment templates with Azure CLI: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-cli

I don't have the time right now to implement this workaround since it's a big change. Anyone who succeed with this workaround?

Actually the supports suggested me to implement the retry task in the pipeline (https://docs.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-195-update#automatic-retries-for-a-task). It is worth a try. In the last deployment the resources came consistently available after 20 hours

I too have this issue with my release pipelines. Any recent updates?

@adalfa What is the preview feature?

For what I've understood they changing the storage of the metadata

I'm facing this issue in one of my new pipelines I setup today. Previously deployed image worked and app is even served, CLI finds the web app service. Pipeline intermittently fails deploy with the above error message even though the app service exists.

For what it's worth, I changed my deployment Tasks to "Run From Package". That seemed to help, but obviously it's not a solution for everyone.

Same issue here detected on 2021-Q1. I opened a support ticket with Microsoft and they contact me to see the error in place. After many attempts for some reason they told me to change the version of "Azure App Service deploy" task. I had 4.* and they suggested me to set 2.*. That got the pipeline working fine, but about three months later the problem reappeared and got worse because it started taking hours for "Azure App Service deployment" task to find the already created App Service.
I'm now using the suggested solution mentioned in this thread which is to use the "Azure CLI" task to do the deployment. Meanwhile I'm waiting for this issue to be resolved before going back to the "Azure App Service Deployment" task solution, which is a cleaner solution.

We are still experiencing this issue.
We created an azure function app via terraform, and we are attempting to use the AzureFunctionApp@1 task to deploy our function app.
The ADO pipeline either throws resource does not exist error, or hangs for 30 minutes before throwing Failed to deploy web package to app service. Service Unavailable (CODE: 503) error.
az resource list successfully returns the function app resource as well.

For us situation was different. All the functions (linux, dotnet-isolated) resources are provisioned by pulumi script.
For Dev/Statging env SKU is Y1 and linuxFxVersion is DOTNET|6.0. On this setup, everything is fine and working.
Prod env SKU is EP1 and with linuxFxVersion DOTNET|6.0 functions can not create host.json or keys. So we had to set linuxFxVersion as DOTNET-ISOLATED|6.0. Anyone having a similar situation, this may help.

https://github.com/microsoft/azure-pipelines-tasks/issues/15532

We have a similar error:
2022-06-24T00:47:17.6400048Z Got service connection details for Azure App Service:'app-client-staging-stg'
2022-06-24T00:47:18.8535752Z (node:5656) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined
2022-06-24T00:47:18.8536807Z     at WindowsWebAppRunFromZipProvider. (D:\a_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\4.205.13\deploymentProvider\WindowsWebAppRunFromZipProvider.js:68:77)
2022-06-24T00:47:18.8537582Z     at Generator.next ()
2022-06-24T00:47:18.8538270Z     at fulfilled (D:\a_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\4.205.13\deploymentProvider\WindowsWebAppRunFromZipProvider.js:5:58)
2022-06-24T00:47:18.8539029Z     at process._tickCallback (internal/process/next_tick.js:68:7)
2022-06-24T00:47:18.8540017Z (node:5656) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
2022-06-24T00:47:18.8541120Z (node:5656) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2022-06-24T00:47:18.8574131Z ##[error]Error: Resource 'app-client-staging-stg' doesn't exist. Resource should exist before deployment.
But the resource exists. When I rerun this stage, it works. We are using one pipeline to deploy everything. Ultimately, it tries to deploy 2 web services, 1 API service, and 1 function app. And it always fails on the web services (randomly).

Yup, still the issue for a long time. Had a support case with MS, but everything that they suggested is to create 'a wait job' for ~25-40min to allow sync metadata between regions.
Also they replied me with this:
I will share with you a summary from the ARM perspective: I informed you that the issue is being caused by a delay in the replication flow of resource metadata between multiple Azure regions and our product team is aware of the issue and they are testing a possible solution for the issue; however, currently there is not an ETA for the fix of the issue. You also asked if the is any public documentation regarding this issue, I informed you that our product teams has not published any public documentation about this, since they want to ensure that the fix can be applied properly across all regions.
Because I create many similar environments from scratch daily (use automation templates), that error is a catastrophe. All the time, we face this error, and even a 30-40 min delay after the creation of resources does not guarantee that deployment will be successful. Moreover, a validate task (az cli, REST API, etc.) prior AzureRMWebAppDeployment task shows that resource is up and running, but AzureRMWebAppDeployment does not see this. Retry option (possible to do max ten times) for ARM  task also does not help a lot.  Yeah, that delay and retry decreased failed jobs, but it did not fix the issue.
Hope this will be fixed asap :-/

I've managed to fix it with the workaround provided by Microsoft: we are now using the Azure CLI for deploying webapps and the Azure Functions tools for deploying Azure Functions.
Old app service deployment:
  - download: "current"
    displayName: "Download UI.Api"
    artifact: "UI.Api"
  - task: AzureRmWebAppDeployment@4
    displayName: "Deploy: UI.Api"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      ConnectionType: "AzureRM"
      azureSubscription: "$(azureResourceManagerConnection)"
      appType: "webApp"
      WebAppName: "app-api-${{ parameters.projectName }}-${{ parameters.environment }}"
      packageForLinux: "$(Pipeline.Workspace)/UI.Api/"
      enableCustomDeployment: true
      DeploymentType: "zipDeploy"
      enableXmlVariableSubstitution: true
      deployToSlotOrASE: ${{ parameters.useDeploymentSlots }}
      SlotName: "staging"
      resourceGroupName: "${{ parameters.resourceGroup }}"

New app service deployment:
  - download: "current"
    displayName: "Download UI.Api"
    artifact: "UI.Api"
  - task: FileTransform@1
    displayName: "Transform UI.Api"
    inputs:
      folderPath: "$(Pipeline.Workspace)/UI.Api/UI.Api.zip"
      fileType: "xml"
      targetFiles: "Web.config"
  - task: AzureCLI@2
    displayName: "Deploy UI.Api"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      azureSubscription: $(azureResourceManagerConnection)
      scriptType: ps
      scriptLocation: inlineScript
      inlineScript: |
        if ("${{ parameters.useDeploymentSlots }}" -eq "true") {
          az webapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings WEBSITE_RUN_FROM_PACKAGE="1" `
          --slot "staging"

          az webapp deployment source config-zip `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --src "$(Pipeline.Workspace)/UI.Api/UI.Api.zip" `
          --slot "staging"
        }
        else {
          az webapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings WEBSITE_RUN_FROM_PACKAGE="1"

          az webapp deployment source config-zip `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --src "$(Pipeline.Workspace)/UI.Api/UI.Api.zip"
        }

Old Azure Function deployment:
  - download: "current"
    displayName: "Download UI.Function"
    artifact: "UI.Function"
  - task: AzureFunctionApp@1
    displayName: "Deploy UI.Function"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      azureSubscription: "$(azureResourceManagerConnection)"
      appType: "functionApp"
      appName: "app-function-${{ parameters.projectName }}-${{ parameters.environment }}"
      package: "$(Pipeline.Workspace)/UI.Function"
      deploymentMethod: "zipDeploy"
      resourceGroupName: "${{ parameters.resourceGroup }}"
      appSettings: '-AzureDevopsToken "${{ parameters.azureDevopsToken }}" -EnvironmentName "${{ parameters.environment }}"'

New Azure Function deployment:
  - task: NodeTool@0
    displayName: "Use Node 14"
    inputs:
      versionSpec: 14.18.1
  - task: Npm@1
    displayName: "Install Azure Functions tools"
    inputs:
      command: "custom"
      customCommand: "install -g azure-functions-core-tools"
  - download: "current"
    displayName: "Download UI.Function"
    artifact: "UI.Function"
  - task: ExtractFiles@1
    displayName: "Extract UI.Function"
    inputs:
      archiveFilePatterns: "$(Pipeline.Workspace)/UI.Function/UI.Function.zip"
      destinationFolder: "$(Pipeline.Workspace)/Extracted/"
      cleanDestinationFolder: true
      overwriteExistingFiles: false
  - task: AzureCLI@2
    displayName: "Deploy UI.Function"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      workingDirectory: "$(Pipeline.Workspace)/Extracted/"
      azureSubscription: $(azureResourceManagerConnection)
      scriptType: pscore
      scriptLocation: inlineScript
      inlineScript: |
        az functionapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-function-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings AzureDevopsToken="${{ parameters.azureDevopsToken }}" EnvironmentName="${{ parameters.environment }}"

        func azure functionapp publish app-function-${{ parameters.projectName }}-${{ parameters.environment }} `
          --typescript

Some notes:

Release annotations in ApplicationInsight is not automatically added by the Azure CLI.
I recommend to still add and retryCountOnTaskFailure of minimal 2. It seems the deployment can be still flaky with newly created resources.

@maikelvanhaaren: This "solution" with inline script and Azure CLU if really difficult to maintain in long term (1 year+)

@Saibamen I rarely need to change the deployments of the apps and I guess that's the same for (almost) every release-pipeline.

Ok, but you know that API and CLI can change without your knowledge ;)

I guess that's the same potential issue you will face for using every external tool/SDK/API..

@maikelvanhaaren thank you for sharing. Yeah, also thought about such tasks. I did not want to use a custom task for deployment, but it is worth trying as a workaround to make life a little easier finally. The bad thing is that now I need to explain to support details why I use bicycles like this instead of an integrated feature. :D

I'm also experiencing this issue and found that it's likely due to the encoding of the character '/' in the $filter. When I manually try the endpoint in the logs it results in an empty response body whereas using the actual '/' character returns the expected result
Pipeline task URL
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'hg-identity-web-dev-eastus2'&api-version=2016-07-01
{
  "value": []
}

Manual written URL
https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'hg-identity-web-dev-eastus2'&api-version=2016-07-01
{
  "value": [
    {
      "id": "/subscriptions/.../resourceGroups/rg-identity-service-dev/providers/Microsoft.Web/sites/hg-identity-web-dev-eastus2",
      "name": "hg-identity-web-dev-eastus2",
}

It looks like @johnmccormick99 may have discovered an intriguing workaround for this issue.  We're not certain this is going to work forever, but it sure seems to work right now.  Details in this PR https://github.com/johnnyreilly/blog.johnnyreilly.com/pull/291
You can see the workaround in play here: https://thankful-sky-0bfc7e803-291.westeurope.1.azurestaticapps.net/2022/09/30/azure-functions-resource-should-exist-before-deployment
We're thinking of putting this out as a blog post, but it'd be awesome to get more info from others about whether this works.  Keen to spread god info not bad!

One caveat to the blog post/workaround...  Simply doing the GET on the resource does not "force" it to replicate the metadata... so any number of those calls could still fail.  If you wanted to perfect it, you could put retries on errors in each region.
I suspect you're seeing it work in practice because of the delay created by the task...  global replication is typically pretty fast, though I think this may be slightly different than simple replication.  In any case, if you wanted to test that theory, you could try a "sleep" for the amount of time it takes that task to finish and see how reliable that is...

Thanks for feeding back @bmoore-msft!

If you wanted to perfect it, you could put retries on errors in each region.

I just wanted to make sure I understand you correctly here.  Where we are doing the following:
      $locations | foreach-Object -Parallel {
          $restUrl = "https://$_.management.azure.com/subscriptions/$(subscriptionId)/resourceGroups/$(azureResourceGroup)/providers/Microsoft.Web/sites/$(namerWapiFunctionApp)?api-version=2022-03-01";
          ((((Invoke-WebRequest -Uri $restUrl -Method GET -Headers $($using:authHeader)).Headers).GetEnumerator() | Where-Object {$_.Key -ieq "x-ms-routing-request-id"}).Value).Split(":")[0]
      }

What you're saying is, that call could 404 / error, and to make it more resilient you could check whether we're getting a 404 / error back and re-attempt the call?

I suspect you're seeing it work in practice because of the delay created by the task... global replication is typically pretty fast, though I think this may be slightly different than simple replication. In any case, if you wanted to test that theory, you could try a "sleep" for the amount of time it takes that task to finish and see how reliable that is...

So I think we may have already travelled this road. We've been implemented sleep tasks across multiple pipelines in my organisation for for some time.  We typically sleep for 30 seconds.  It marginally improves reliability, but the key word in there is "marginally".  The task we've been running above typically takes around 12 seconds to run.  Whilst we've only been testing this for a day now, we've yet to see a failed deployment.
It may be entirely coincidental; perhaps for Azure yesterday was a good day [resists the temptation to insert a GIF of ICE Cube].  We're certainly going to keep testing this in our organisation and see how it plays out.
This doesn't feel like it ought to work, but it does seem to!
Related?
Oh one other thing to feed into this that may be related.  About a year ago when we were trying various things to improve the resilience of our Azure Function deployments we pivoted our approach based upon some discoveries by @jamiemccrindle.
We found that instead of using the AzureWebApp@1 for deployment, we could use the AZ command line which was less prone to failure.   So instead of:
    - task: AzureWebApp@1
      displayName: 'Deploy Function'
      retryCountOnTaskFailure: 3 # retryCountOnTaskFailure has improved the reliability of our pipelines no end
      inputs:
        azureSubscription: ${{ variables.serviceConnection }}
        appName: $(maasMscopeFunctionApp)
        package: $(Pipeline.Workspace)/MaaS.MicroscopeFunctionApp/MaaS.MicroscopeFunctionApp.zip
        resourceGroupName: $(resourceGroup)
        deployToSlotOrASE: true
        slotName: stage

We would use:
    - task: AzurePowerShell@5
      displayName: 'Deploy Function'
      retryCountOnTaskFailure: 3
      inputs:
        azureSubscription: ${{ variables.serviceConnection }}
        azurePowerShellVersion: 'LatestVersion'
        ScriptType: 'InlineScript'
        Inline: |
          Publish-AzWebApp -ResourceGroupName $(resourceGroup) -Name $(maasMscopeFunctionApp) -Slot stage -ArchivePath $(Pipeline.Workspace)/MaaS.MicroscopeFunctionApp/MaaS.MicroscopeFunctionApp.zip -Force

Which would fail less often.  I forget the details now, but I believe when @jamiemccrindle did some digging, he was able to confirm that the AzureWebApp@1 task relied upon a different API to Publish-AzWebApp for testing if a function was there.   Alas I don't recall the details.
As you say "global replication is typically pretty fast".  This our experience also, we think the "Resource should exist before deployment" is actually evidence of problematic cache invalidation.  The cache doesn't realise that a function is deployed.

I just wanted to make sure I understand you correctly here. Where we are doing the following:
      $locations | foreach-Object -Parallel {
          $restUrl = "https://$_.management.azure.com/subscriptions/$(subscriptionId)/resourceGroups/$(azureResourceGroup)/providers/Microsoft.Web/sites/$(namerWapiFunctionApp)?api-version=2022-03-01";
          ((((Invoke-WebRequest -Uri $restUrl -Method GET -Headers $($using:authHeader)).Headers).GetEnumerator() | Where-Object {$_.Key -ieq "x-ms-routing-request-id"}).Value).Split(":")[0]
      }

What you're saying is, that call could 404 / error, and to make it more resilient you could check whether we're getting a 404 / error back and re-attempt the call?

Yes - so for each endpoint you query - you may get a 404 if replication hasn't hit that region yet... so if you did a retry/loop until you got a 200, that would be about the best you could do to check replication.

So I think we may have already travelled this road. We've implemented sleep tasks across multiple pipelines in our organisation over the last year or two. We typically sleep for 30 seconds. It marginally improves reliability, but the key word in there is "marginally". The task we've been running above typically takes around 12 seconds to run. Whilst we've only been testing this for a day now, we've yet to see a failed deployment afterwards.

More on this below...

It may be entirely coincidental; perhaps for Azure, yesterday was a good day [resists the temptation to insert a GIF of ICE Cube]. We're certainly going to keep testing this in our organisation and see how it plays out.

And I kinda want to see the GIF now...

Which would fail less often. I forget the details now, but I believe when @jamiemccrindle did some digging, he was able to confirm that the AzureWebApp@1 task relied upon a different API to Publish-AzWebApp for testing if a function was there. It's been a while but he thinks the AzureWebApp@1 task depended upon the Search API / Resource Manager API (can't remember) to find the function. The joke being that whichever API it used, that API put the "eventually" into "eventually consistent" 😄

I don't know a ton about what functions look like on the back end, but if publishing via that command uses a different API you could certainly get different results if you do this in quick succession.  One way to check is to throw the -debug switch on those command and see what endpoint it hits (if you're bored/interested).

@bmoore-msft, as you say "global replication is typically pretty fast". This our experience also, we think the "Resource should exist before deployment" is actually evidence of problematic cache invalidation. The cache doesn't realise that a function is deployed and returns false information.

There are certainly outlier days - but the think that makes me think there's more going on here is the error message Resource should exist before deployment - that's not a typical error message you would see if it was a replication error (that's just a 404/not found).  So there may be more going on behind the scenes with this particular case.  Put another way, if it's not simply due to replication, the script to check each endpoint could still have issues - i.e. it could have all 200s but still encounter that particular error.  That's just a guess...  I'll see if I can dig up any more detail on that error.

Azure when we tested @johnmccormick99's workaround:

There are certainly outlier days - but the think that makes me think there's more going on here is the error message Resource should exist before deployment - that's not a typical error message you would see if it was a replication error (that's just a 404/not found).  So there may be more going on behind the scenes with this particular case.  Put another way, if it's not simply due to replication, the script to check each endpoint could still have issues - i.e. it could have all 200s but still encounter that particular error.  That's just a guess...  I'll see if I can dig up any more detail on that error.

Error is generated from the code here:

https://github.com/microsoft/azure-pipelines-tasks/blob/058d7a428080b089cf50473960839972d3b60612/common-npm-packages/AzureRmDeploy-common-v3/operations/AzureResourceFilterUtility.ts#L11-L13
https://github.com/microsoft/azure-pipelines-tasks/blob/058d7a428080b089cf50473960839972d3b60612/common-npm-packages/AzureRmDeploy-common-v3/azure-arm-rest/azure-arm-resource.ts#L39-L40

We ran into the same issue (EU West).
Once it took about an hour to resolve itself, another time about a day.
We tried some things on our end to fix it, but nothing seems to have helped: It seems to have magically resolved itself

Wow, it's hitting a year now, with this issue still unresolved.

Just experienced this error on a new function app as well (test stage) - been trying the deployment for 2 hours with no luck. Has been working fine on dev stage

We are still experiencing this issue for our Terraform/Devops pipelines. Workarounds are not 100% reliable, delays do work, but are increasing the duration of our pipelines by a lot. Resource creation and deployment via local machine with user credentials does not seem to have this problem.

It's 2023 and after 1 year of the first post, this hasn't yet been resolved. In our case, we are trying to move resources from one region to another. Functions don't have the "Move" option and therefore we need to delete it, re-create it and then go into DevOps to redeploy the code.
I understand the existing/old Pipeline (in Azure DevOps) will fail because the resource (ID) changed. Here's what I usually try:
1 - Refresh the existing Steps to force a Resource ID update (the dropdowns)
2 - Manually type in the name of the Resource in the Application dropdown (because, of course, it's not there)
3 - Create new pipelines to force a whole process
4 - Get the Service Principal and give Read permission under the Resource Group and the Resource itself
5 - Refresh the Service Connection inside the project
Nothing, I repeat, NOTHING guarantees that the next pipeline run will work. What works is just "wait for a random period and" try again.
I sincerely don't understand why this is still a thing.

Hi Guys, Just hit the issue today. Will it be fixed?

Hi Guys! I've commented here long time ago. This issue basically roots in the inconsistency of the distributed storage in which Azure stores our resources metadata. When we create/update resources, they metadata initially gets stored in one (or at least, not in all the) location and as time goes it gets replicated to multiple metadata stores. Because of this delay, we can reach "ARM endpoints" which do not know about our resource creation/update yet. So it can happen that with your first call you are creating a resource which's metadata is stored in location A, and as a next step you try to deploy on it from Azure Pipelines (for example) but that deployment checks the existence of your resource in location B which simply does not contain your previous updates yet.
To overcome this challenge, we opened a support ticket in Azure Portal and it it turned out that we were eligible to migrate our Tenant's ARM (storage ?) to use CosmosDB. Since its replication latency was way more lower than the original solution's this solved our issue immediately.
I'm not sure I used the correct wording everywhere, but I'm sure you got the ideas :)

I have just started getting this today - its really infuriating.  My app did deploy once, but has failed before and since.  I just created a brand new App Service and I am getting the same error on that (never seen it before on hundreds of others I have created over the years)
Got service connection details for Azure App Service:'RevelationMoonAdminNew'
##[debug][POST]https://login.windows.net/b02e6e0c-de2b-465f-ba6a-88d7c375af11/oauth2/token/
##[debug][GET]https://management.azure.com/subscriptions/a3ffcdfc-2958-4362-8ec6-259dbf71229f/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'RevelationMoonAdminNew'&api-version=2016-07-01
##[debug]Correlation ID from ARM api call response : 448d8eb9-f235-4cb6-8aea-875f4c883152
##[debug]Deployment Failed with Error: Error: Resource 'RevelationMoonAdminNew' doesn't exist. Resource should exist before deployment.
##[debug]task result: Failed
##[error]Error: Resource 'RevelationMoonAdminNew' doesn't exist. Resource should exist before deployment.
##[debug]Processed: ##vso[task.issue type=error;]Error: Resource 'RevelationMoonAdminNew' doesn't exist. Resource should exist before deployment.
##[debug]Processed: ##vso[task.complete result=Failed;]Error: Resource 'RevelationMoonAdminNew' doesn't exist. Resource should exist before deployment.
##[debug]Deployment failed

@borondy  same option the support gave use last year, I think Azure should make the feature GA...

Same issue here. Between this issue and the intermittent 'service unavailable (code 503)' when it tries to mount the zip, function app deployments are painful right now.

To overcome this challenge, we opened a support ticket in Azure Portal and it it turned out that we were eligible to migrate our Tenant's ARM (storage ?) to use CosmosDB. Since its replication latency was way more lower than the original solution's this solved our issue immediately.

Sounds like this should be how it works behind the scene by default. At its current state, the Azure iac pipeline implementation is unstable and simply just broken. I wonder why Microsoft is sleeping on this?

In the end it turns out the CLI is more reliable. So we used the solution previously mentioned.

I've managed to fix it with the workaround provided by Microsoft: we are now using the Azure CLI for deploying webapps and the Azure Functions tools for deploying Azure Functions.
Old app service deployment:
  - download: "current"
    displayName: "Download UI.Api"
    artifact: "UI.Api"
  - task: AzureRmWebAppDeployment@4
    displayName: "Deploy: UI.Api"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      ConnectionType: "AzureRM"
      azureSubscription: "$(azureResourceManagerConnection)"
      appType: "webApp"
      WebAppName: "app-api-${{ parameters.projectName }}-${{ parameters.environment }}"
      packageForLinux: "$(Pipeline.Workspace)/UI.Api/"
      enableCustomDeployment: true
      DeploymentType: "zipDeploy"
      enableXmlVariableSubstitution: true
      deployToSlotOrASE: ${{ parameters.useDeploymentSlots }}
      SlotName: "staging"
      resourceGroupName: "${{ parameters.resourceGroup }}"

New app service deployment:
  - download: "current"
    displayName: "Download UI.Api"
    artifact: "UI.Api"
  - task: FileTransform@1
    displayName: "Transform UI.Api"
    inputs:
      folderPath: "$(Pipeline.Workspace)/UI.Api/UI.Api.zip"
      fileType: "xml"
      targetFiles: "Web.config"
  - task: AzureCLI@2
    displayName: "Deploy UI.Api"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      azureSubscription: $(azureResourceManagerConnection)
      scriptType: ps
      scriptLocation: inlineScript
      inlineScript: |
        if ("${{ parameters.useDeploymentSlots }}" -eq "true") {
          az webapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings WEBSITE_RUN_FROM_PACKAGE="1" `
          --slot "staging"

          az webapp deployment source config-zip `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --src "$(Pipeline.Workspace)/UI.Api/UI.Api.zip" `
          --slot "staging"
        }
        else {
          az webapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings WEBSITE_RUN_FROM_PACKAGE="1"

          az webapp deployment source config-zip `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-api-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --src "$(Pipeline.Workspace)/UI.Api/UI.Api.zip"
        }

Old Azure Function deployment:
  - download: "current"
    displayName: "Download UI.Function"
    artifact: "UI.Function"
  - task: AzureFunctionApp@1
    displayName: "Deploy UI.Function"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      azureSubscription: "$(azureResourceManagerConnection)"
      appType: "functionApp"
      appName: "app-function-${{ parameters.projectName }}-${{ parameters.environment }}"
      package: "$(Pipeline.Workspace)/UI.Function"
      deploymentMethod: "zipDeploy"
      resourceGroupName: "${{ parameters.resourceGroup }}"
      appSettings: '-AzureDevopsToken "${{ parameters.azureDevopsToken }}" -EnvironmentName "${{ parameters.environment }}"'

New Azure Function deployment:
  - task: NodeTool@0
    displayName: "Use Node 14"
    inputs:
      versionSpec: 14.18.1
  - task: Npm@1
    displayName: "Install Azure Functions tools"
    inputs:
      command: "custom"
      customCommand: "install -g azure-functions-core-tools"
  - download: "current"
    displayName: "Download UI.Function"
    artifact: "UI.Function"
  - task: ExtractFiles@1
    displayName: "Extract UI.Function"
    inputs:
      archiveFilePatterns: "$(Pipeline.Workspace)/UI.Function/UI.Function.zip"
      destinationFolder: "$(Pipeline.Workspace)/Extracted/"
      cleanDestinationFolder: true
      overwriteExistingFiles: false
  - task: AzureCLI@2
    displayName: "Deploy UI.Function"
    retryCountOnTaskFailure: ${{ parameters.retryDeployCount }}
    inputs:
      workingDirectory: "$(Pipeline.Workspace)/Extracted/"
      azureSubscription: $(azureResourceManagerConnection)
      scriptType: pscore
      scriptLocation: inlineScript
      inlineScript: |
        az functionapp config appsettings set `
          --resource-group "${{ parameters.projectName }}-${{ parameters.environment }}" `
          --name "app-function-${{ parameters.projectName }}-${{ parameters.environment }}" `
          --settings AzureDevopsToken="${{ parameters.azureDevopsToken }}" EnvironmentName="${{ parameters.environment }}"

        func azure functionapp publish app-function-${{ parameters.projectName }}-${{ parameters.environment }} `
          --typescript

Some notes:

Release annotations in ApplicationInsight is not automatically added by the Azure CLI.
I recommend to still add and retryCountOnTaskFailure of minimal 2. It seems the deployment can be still flaky with newly created resources.

I got the same issue when deloying to the new-created App Service. I had to wait for 2 days and try again, now the name of that App Service appears in the App Service name of Azure App Service deploy task (version: 4.*) in the Release pipeline.

I am getting the same issue still with web app deploys.  Going to try to implement the CLI workaround, but it is a bit disappointing that Azure still has this issue.

I was getting this error with deploying to a new App Service with v4 ( I tried several times ). It worked with v3, after which I was immediately able to switch back to v4 and it worked.

Because it is a nasty bug, nothing more.
czw., 18 maj 2023 o 12:22 Greig Stewart @.***>
napisał(a):

I was getting this error with deploying to a new App Service with v4 ( I
tried several times ). It worked with v3, after which I was immediately
able to switch back to v4 and it worked.
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/azure-pipelines-tasks/issues/15532#issuecomment-1552847464,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIAUSZ4WAY4X47FQIMWDD63XGXZ77ANCNFSM5ISNBOOA
.
You are receiving this because you commented.Message ID:
@.***>

Yesterday I faced the below error when deploying function app via Azure Devops yaml pipeline:
##[error]Error: Resource 'fn-app1' doesn't exist. Resource should exist before deployment.

Interestingly, another pipeline with the same steps worked fine. The only difference was, different service principals (ADO service connections) were used in the two pipelines.
After comparing the service principals I found that mine had RBAC permissions on multiple subscriptions (including the function app subscription) where as the working one had permissions on only one subscription containing function app.
The Fix was simple - I created a new service principal with permissions to only the function app and related resources in same subscription. So "az account show" gives only one subscription in the output.
Hope this helps someone.

Any update? Lately it started becoming more and more unstable. I've also tried having the reset azure cache suggested, added 10 retries on the task and it is still failing.

Any update?
I'm a teacher and was teaching Azure DevOps Pipelines to my students using this code and tutorial from mslearn and we are experimenting the same issue... It's very frustrating, as students don't know if they're configuring something wrong or there is any other problem.
It's the same issue described on nov 2021... I also tried adding delays between ARM taks and deploy task but it continues failing ramdonly.
The code students are using is here
For specifying the pipelines we are using the classic editor (as it's the first time they are configuring pipelines)

The deployment.json is here
https://github.com/ETSISI-EMS/mslearn-tailspin-spacegame-web/blob/main/azuredeploy.json
The error of deploy task is as follows:
Error: Resource 'webappw0140' doesn't exist. Resource should exist before deployment.
Unhandled: Cannot read properties of undefined (reading 'getApplicationURL')
TypeError: Cannot read properties of undefined (reading 'getApplicationURL')
at WindowsWebAppRunFromZipProvider. (D:\a_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\4.235.0\deploymentProvider\WindowsWebAppRunFromZipProvider.js:68:77)
at Generator.next ()
at fulfilled (D:\a_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\4.235.0\deploymentProvider\WindowsWebAppRunFromZipProvider.js:5:58)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

also it would make sense that it would fail the first time the resources are created because it needs some delay but once the resources are created, why does the deploy fail? also why does it fail randomly?

Starting seeing this issue of "resource not found" since this weekend, probably about the same timeframe as @jdiazfernandez . Pipeline exists for more than a year, all deployments successful, until Sunday... no real means to troubleshoot, as 'it just happened'.
We're deploying using Azure CLI scripts, not the WebApp deployment task.

Didn't change anything in the pipeline definition, but it started working again today. for both WebApp deployments as well as Functions. Hopefully it is stabilized again...

In my case the issue was caused by insufficient permissions. I had to assign Website Contributor role to the service principal.

It is very likely that permissions had nothing to do with it; but rather while you were adjusting permissions, Azure replication completed.

Nothing to do with permissions. I had this issue randomly in the past

My advise to every one - the main reason for this errors is that make sure you have the right role as Website Contributor  to the registered application. Subscriptions


PiranhaSwift: Use properties.json file to specify configuration

Currently PiranhaSwift APIs are specified using a properties file as shown here.  It will be helpful to migrate this to use a json file so that more properties can be specified easily. For instance, it could be an array of methodProperties where the methodName, flagType, returnType and argumentIndex of the flag are specified. Example:
{ "methodProperties": [ { "methodName": "isToggleEnabled", "flagType": "treated", "returnType": "boolean", "argumentIndex": 0 }, ... ], }
This properties file will be parsed and the listed APIs will be used for refactoring.

I can take this up 👍🏼

I can take this up 👍🏼

thanks. assigned it to you.

Fixed in #118