How to correctly delete folders with mixed contents by prefix?

Options
nk9
nk9 Member Posts: 11
edited September 2023 in Developer APIs

I have a bunch of test folders with both images and raw files in them which I want to delete. I have this code:

    let folders = await cloudinary.api.sub_folders('albums').then(resp => resp.folders)
    var folders_to_delete = folders.filter((f) => f.name.startsWith('test'))

    console.log("folders to delete: ", folders_to_delete)

    for (const folder of folders_to_delete) {
        console.log("prefix: ", folder)
        cloudinary.api
            .delete_resources_by_prefix(folder.path, { resource_type: "image" })
            .then(images_result => {
                console.log(images_result);

                cloudinary.api.delete_resources_by_prefix(folder.path,
                    { resource_type: "raw" })
            })
            .finally(info_result => {
                console.log(info_result)
                
                // Once all the folder's contents are gone, then delete the folder itself
                cloudinary.api
                    .delete_folder(folder.path)
                    .then(folder_result => console.log(folder_result))
                    .catch(err => console.log("ERROR: ", err))
            })

    }

But I'm not sure if this is the clearest/most correct way to do what I want. Also, I now have 4 folders which appear to be empty in the Media Library but which the API is refusing to delete because "Folder is not empty." Might this have something to do with derived assets? And regardless, how do I convince the API to actually delete these empty folders?

Tagged:

Answers

  • nk9
    nk9 Member Posts: 11
    Options

    I've got something a bit clearer now, but it's still not deleting the supposedly empty folders. I've even turned on skip_backup, but that hasn't helped.

        let types = ["image", "raw"]
    
        for (const type of types) {
            let resp = await cloudinary.api.resources({
                resource_type: "image",
                max_results: 500,
                prefix: "albums/test",
                type: "upload"
            })
            let public_ids = resp.resources.map(r => r.public_id)
            cloudinary.api.delete_resources(public_ids, { resource_type: type })
        }
    
        let folders = await cloudinary.api.sub_folders('albums').then(resp => resp.folders)
        var folders_to_delete = folders.filter((f) => f.name.startsWith('test'))
    
        console.log("folders to delete: ", folders_to_delete)
    
        for (const folder of folders_to_delete) {
            // Once all the folder's contents are gone, then delete the folder itself
            cloudinary.api
                .delete_folder(folder.path, { skip_backup: true })
                .then(folder_result => console.log(folder_result))
                .catch(err => console.log("ERROR: ", err))
        }
    

    I appreciate that this will only delete the first 500 resources found, and would fail if you subsequently tried to delete folders which still have resources inside. Those limitations are fine, but I do want the empty folders to actually go away…

  • nk9
    nk9 Member Posts: 11
    edited September 2023
    Options

    I have something a bit clearer now, and I've tried using skip_backup to delete folders even if there are backed up resources hiding. Unfortunately, it's still not deleting the empty folders:

        let types = ["image", "raw"]
    
        for (const type of types) {
            let resp = await cloudinary.api.resources({
                resource_type: "image",
                max_results: 500,
                prefix: "albums/test",
                type: "upload"
            })
            let public_ids = resp.resources.map(r => r.public_id)
            cloudinary.api.delete_resources(public_ids, { resource_type: type })
                .then(res => console.log("delete resources success:", res))
        }
    
        let folders = await cloudinary.api.sub_folders('albums').then(resp => resp.folders)
        var folders_to_delete = folders.filter((f) => f.name.startsWith('test'))
    
        console.log("folders to delete: ", folders_to_delete)
    
        for (const folder of folders_to_delete) {
            // Once all the folder's contents are gone, then delete the folder itself
            cloudinary.api
                .delete_folder(folder.path, { skip_backup: true })
                .then(folder_result => console.log(folder_result))
                .catch(err => console.log("ERROR: ", err))
        }
    

    I understand that this would only remove the first 500 resources returned, and would fail if it then tried to delete a non-empty folder. That limitation is fine, but I do still need the empty folders to actually be deleted…

  • nk9
    nk9 Member Posts: 11
    edited September 2023
    Options

    Here is part of the response I'm getting back from delete_resources:

    delete resources success: {
      deleted: {
        'albums/test1/cabinet_handle': 'deleted',
        'albums/test1/fBurma-002': 'deleted',
        'albums/test1/intro_cgi_with_caption': 'deleted',
        'albums/test4/fBurma-011': 'deleted',
    ...
      deleted_counts: {
        'albums/test1/cabinet_handle': { original: 1, derived: 0 },
        'albums/test1/fBurma-002': { original: 1, derived: 0 },
        'albums/test1/intro_cgi_with_caption': { original: 1, derived: 0 },
        'albums/test4/fBurma-011': { original: 1, derived: 0 },
    

    It says the files are deleted, but then it won't delete the folders containing them. And when I run the code again, without having uploaded anything else, the same files are always returned from the api.resources() call. And many of these aren't even in the existing folders. For example, "test1" doesn't show up in the Media Library, but "test4" is there and empty.

    I'm really confused at the way Cloudinary is treating deleted files.

  • Cloudinary_John_H
    Cloudinary_John_H Cloudinary Staff Posts: 36
    Options

    Hi there,

    Thanks for reaching out, and for all of the details.

    This behavior is likely related to having records of backed-up assets in the folders you are trying to delete. When assets have been backed up to the folder, the folder will not be deletable unless the backups are cleared. We do not expose the deletion of backups to end users, in order to help prevent the accidental or intentional malicious deletion of backups.

    Can you share with me the complete list of folder names that are not deleting due to the records of backed-up assets being present?

    Is this the entire list of folders?

     ['albums/test1/','albums/test4']
    

    In order to delete these assets, we will need to assist with the clearing of asset backups associated with these folders. Can you share with me your cloud name please, so I can confirm my suspicions about records of backed-up assets being present in these folders?

    Looking forward to your response,

    John

  • nk9
    nk9 Member Posts: 11
    Options

    Thanks for the reply! Unfortunately, I don't have a complete list of the folders that aren't deleting. I turned off backups at some point, but it was long after I created a bunch of folders in the process of testing. It would be all the ones which start with "test". Anyway, I've opened a support ticket about this just now. Hopefully it will be possible to remove the test*/** backups.

  • Cloudinary_John_H
    Cloudinary_John_H Cloudinary Staff Posts: 36
    Options

    Sounds good - we will assist you further in the support ticket, thank you!

  • nk9
    nk9 Member Posts: 11
    edited September 2023
    Options

    Just to circle back on this, I've decided that it's preferable if I have a way to filter these out, since it's possible to get these sorts of files returned from the resource Admin methods. Filtering out deleted-but-backed-up items is a more futureproof solution vs having to remember never to delete anything, or forcing accounts which use this code to turn off backups.

    I've found that the deleted items have a created_by and uploaded_by property of null. Can you confirm that this is a reliable way to determine whether an asset returned from e.g. search() is deleted or not? And, even better, is there a way to add a predicate to a search which would exclude these from the results before they're returned?

  • Cloudinary_John_H
    Cloudinary_John_H Cloudinary Staff Posts: 36
    Options

    Hey there,

    We provide a field in the Search API expression field called "status", which can be used to filter assets if they are deleted or not.

    https://cloudinary.com/documentation/search_api#expression_fields

    status=deleted
    status=deleted OR active
    
    Possible values: active, deleted
    Notes:
    - Resource data is stored in the database with status deleted only if the resource has a backup.
    - By default, when you conduct a search, the response includes only resources with active status. If you want the response to include resources with a deleted status, you must use this field to specify it.
    Exact match only. Case-sensitive.
    

    Hopefully, that will be useful for you. Based on the description, it sounds like we only return assets that are "active" by default. If you are seeing results that are contrary to that, please let us know the search expression you are using.