Unable to check if asset exists when using "use this filename" option

[jamesmacwhite]

Description

I have a feed which is importing assets to an entry, this is the feed configuration for reference relating to the asset field.

Options configured:

Strategy:

✔Create new Elements
✔Update existing elements

Asset field options:

✔ Create asset from URL
If exists: replace existing asset

Use filename for assets created from URL:
Specific field with filename value: A filename value is provided in the Feed Me mapping.

There's a bit of processing behind the scenes for this asset import initially, because it's coming from a Microsoft SharePoint site (Microsoft Graph API). The image URLs provided directly are not publicly accessible, because it is an intranet site, but there is a way to generate a public image URL with a token that last for 1 hour. This feed is being provided an image URL which is public and valid as it is generated on the feed being triggered, but only for 1 hour. However the idea is, Feed Me can import the asset from this temporary URL when the feed starts and then store this as a asset instead going forward.

The first issue which is unrelated to Feed Me is the URL generated from the Microsoft Graph API looks like this horrible mess:

https://ukwest1-mediap.svc.ms/transform/thumbnail?provider=spo&inputFormat=png&cs=NWE2MWI5M2YtNzBkMy00OGRhLWFiNmUtNGFkOWZkM2UzNjgwfFNQTw&docid=https%3A%2F%2Fexample.sharepoint.com%2F_api%2Fv2.0%2Fdrives%2Fb!wt9_FRAbtEKE0np4yV82bRtcCkE_MnREpOFqIJDiJ3eHCqPq7QJfQqIyMBzSSIqY%2Fitems%2F01LVK66DAN6DS5N7XSFVA2FJWWB335ZNMV%3Ftempauth%3Dv1.eyJzaXRlaWQiOiIxNTdmZGZjMi0xYjEwLTQyYjQtODRkMi03YTc4Yzk1ZjM2NmQiLCJhcHBfZGlzcGxheW5hbWUiOiJOb3R0aW5naGFtIENvbGxlZ2UgQVBJIFNoYXJlUG9pbnQiLCJhcHBpZCI6IjVhNjFiOTNmLTcwZDMtNDhkYS1hYjZlLTRhZDlmZDNlMzY4MCIsImF1ZCI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9ub3R0aW5naGFtY29sbGVnZWFjdWsuc2hhcmVwb2ludC5jb21AMzYwODQ3YTgtZmRmMy00ZGU2LWI3YzgtNDU3MmM5MWM1YjgxIiwiZXhwIjoiMTcyMjYxMDgwMCJ9.CgoKBHNuaWQSAjg0EgYIwrQ6EAEaDTIwLjE5MC4xNTkuMjkqLEt2cjc1dmNmQzhCVy82NmR6TWFXaGk0ZndVT2g0aDlIY3N6bHJGUElac0U9MKkBOAFKEGhhc2hlZHByb29mdG9rZW5SHVsia21zaSIsImR2Y19jbXAiLCJkdmNfZG1qZCJdcikwaC5mfG1lbWJlcnNoaXB8MTAwMzdmZmVhNWYzMDJiOUBsaXZlLmNvbXoBMoIBEgmoRwg28_3mTRG3yEVyyRxbgZIBBUphbWVzmgEFV2hpdGWiASNqYW1lcy53aGl0ZUBub3R0aW5naGFtY29sbGVnZS5hYy51a6oBEDEwMDM3RkZFQTVGMzAyQjmyAR5hbGxzaXRlcy5yZWFkIGFsbHByb2ZpbGVzLnJlYWQ.cDOSv7_A78buJi0A1Oq0GgBBX5-rU9Nul8UjbbZ-SdI%26version%3DPublished&width=800&height=800&cb=63844722231

One of the first major problems is there is no file extension to infer this is an image, which is a problem when importing to image fields with specific file type restrictions. Likewise getting any usable filename from the URL like this offers nothing valuable or remotely usuable.

However, using the filename option, we are providing something that works. The origin API data provides the original filename of the generated image even though this cannot be used as a image URL path, so a value such as "25938-Army-Careers.png" is provided to the "Use this filename for assets created from URL" option, which resolves this problem.

This allows the asset import to work and not get blocked by the lack of file extension, but every time the feed runs the asset import doesn't appear to be working entirely and is logging this error:

newImage is the field handle of the asset field.

newsImage - Asset error: https://ukwest1-mediap.svc.ms/transform/thumbnail?provider=spo&inputFormat=png&cs=NWE2MWI5M2YtNzBkMy00OGRhLWFiNmUtNGFkOWZkM2UzNjgwfFNQTw&docid=https%3A%2F%2Fexample.sharepoint.com%2F_api%2Fv2.0%2Fdrives%2Fb!wt9_FRAbtEKE0np4yV82bRtcCkE_MnREpOFqIJDiJ3eHCqPq7QJfQqIyMBzSSIqY%2Fitems%2F01LVK66DAN6DS5N7XSFVA2FJWWB335ZNMV%3Ftempauth%3Dv1.eyJzaXRlaWQiOiIxNTdmZGZjMi0xYjEwLTQyYjQtODRkMi03YTc4Yzk1ZjM2NmQiLCJhcHBfZGlzcGxheW5hbWUiOiJOb3R0aW5naGFtIENvbGxlZ2UgQVBJIFNoYXJlUG9pbnQiLCJhcHBpZCI6IjVhNjFiOTNmLTcwZDMtNDhkYS1hYjZlLTRhZDlmZDNlMzY4MCIsImF1ZCI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9ub3R0aW5naGFtY29sbGVnZWFjdWsuc2hhcmVwb2ludC5jb21AMzYwODQ3YTgtZmRmMy00ZGU2LWI3YzgtNDU3MmM5MWM1YjgxIiwiZXhwIjoiMTcyMjYxMDgwMCJ9.CgoKBHNuaWQSAjg0EgYIwrQ6EAEaDTIwLjE5MC4xNTkuMjkqLEt2cjc1dmNmQzhCVy82NmR6TWFXaGk0ZndVT2g0aDlIY3N6bHJGUElac0U9MKkBOAFKEGhhc2hlZHByb29mdG9rZW5SHVsia21zaSIsImR2Y19jbXAiLCJkdmNfZG1qZCJdcikwaC5mfG1lbWJlcnNoaXB8MTAwMzdmZmVhNWYzMDJiOUBsaXZlLmNvbXoBMoIBEgmoRwg28_3mTRG3yEVyyRxbgZIBBUphbWVzmgEFV2hpdGWiASNqYW1lcy53aGl0ZUBub3R0aW5naGFtY29sbGVnZS5hYy51a6oBEDEwMDM3RkZFQTVGMzAyQjmyAR5hbGxzaXRlcy5yZWFkIGFsbHByb2ZpbGVzLnJlYWQ.cDOSv7_A78buJi0A1Oq0GgBBX5-rU9Nul8UjbbZ-SdI%26version%3DPublished&width=800&height=800&cb=63844722231 - Unable to check if news/84365-rapincel-WORLD-BOOK-DAY-20_2024-08-02-113032_iapr.png exists.

It appears because of using the replace i.e. conflict option, it couldn't confirm if the image existed. However, the image does exist and was first imported on the very first time this item was imported. Any subsequent update however triggers the same error, across any item.

I originally had "Use existing asset" set, however I found on every feed run it was duplicate the same image but naming it with _xxxx and generated 1000s of images. With "replace existing asset" it appears to be throwing an error but doesn't duplicate images each time.

I'm not sure if there's a bug here, or if due to the exotic asset importing setup needed, it's going outside of the current scope of asset importing/handling.

Steps to reproduce

1. Setup asset field on entry with image type restriction.
2. Setup a feed with importing an asset from a URL which is not very friendly, lacking a file extension and filename.
3. Create asset from URL, if exists "replace existing asset".
4. Configure the use this filename option with a clean value with the file extension.
5. After the first import Feed Me will report Unable to check if the image exists.

Additional info

• Craft version: 5.2.9
• PHP version: 8.2
• Database driver & version: MySQL
• Plugins & versions: Feed Me 6.2.1

[i-just]

Hi, thanks for getting in touch and for all the info!

We already use the value mapped under “Use this filename for assets created from URL” to check if the asset exists: https://github.com/craftcms/feed-me/blob/6.2.1/src/fields/Assets.php#L183-L185. I have tested this, and I can see it working as expected.

In your case, I suspect the problem might be with the concatenation with getRemoteUrlExtension().
Technically, the value under “Use this filename for assets created from URL” should be just the filename without extension. The extension is supposed to be grabbed from the remote file.

That being said, it looks like, in the case of those very special URLs, we cannot determine the extension of the remote file, so providing a value under “Use this filename for assets created from URL” with an extension works as you’d expect it to. That takes care of the first import, but on the second import, assuming the filename from your data is my-image.jpg, the one used to check for existing assets, could be something like my-image.jpg. - this then causes the existing file not to be found and the attempt to create a new one.

That explains why the match is not being found, and I have raised a PR to adjust this check to account for an empty extension.

The Unable to check if <filename> exists. error comes from a flysystem package, not Feed Me, from the fileExists() method, which is supposed to check if a given file exists.

• Could you let me know what filesystem you’re using?
• If it’s not a local filesystem, could you switch to local and see if that changes things?
• Finally, it would be great to know what Feed Me logs say about this. Could you let me know what you see when you click “Show details” for the node that triggered this error?