Why are some of my uploads silently failing the first time or two?

nk9

I am using a Next.js API route (i.e. server-side function) to upload a raw file to a specific location within my Cloudinary account. Here is the code:

import { v2 as cloudinary } from 'cloudinary'
import { Readable } from 'stream';

export default async function uploadInfoJSON(req, res) {
    console.log("enter upload info.json")
    if (req.method !== 'POST') {
        res.status(405).send({ message: 'Only POST requests allowed' })
        return
    }

    cloudinary.config({
        cloud_name: process.env.NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME,
        api_key: process.env.CLOUDINARY_KEY,
        api_secret: process.env.CLOUDINARY_SECRET,
        secure: true
    })

    const { slug } = req.query;
    var upload_stream = cloudinary.uploader.upload_stream(
        {
            public_id: 'info.json',
            folder: `albums/${slug}`, // Passed into the API route as [slug].js
            resource_type: 'raw'
        },
        (error, result) => {
            if (error) {
                console.log("ERROR:", error, result);
            } else {
                console.log("SUCCESS!")
            }
        }
    )
    
    var jsonString = JSON.stringify(req.body, null, 2)
    console.log("JSON=", jsonString)
    Readable.from(jsonString).pipe(upload_stream)

    res.status(200).end()
};

I have a button which the user clicks on a web page, and then a POST request is sent to this route with the JSON as the body. This works fine locally. I click the button, the file is uploaded, and I see "SUCCESS" in the server-side console logging in my terminal.

On Netlify, however, I see this in the "Function Next.js SSR handler" logs:

Sep 5, 09:51:29 PM: INIT_START Runtime Version: nodejs:18.v11 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:2d38a43033930b7e89d977b60a6556a2e84bcc276ff59eab90d1327b252528ef
Sep 5, 09:51:30 PM: 64f72c4c INFO enter upload info.json
Sep 5, 09:51:30 PM: 64f72c4c INFO JSON= {
"display_name": "Test 3",
"description": "",
"cover_photo": "photo3",
"captions": {
"photo1": "caption 1",
"photo2": "caption 2",
"photo3": "caption 3"
},
"is_public_album": true
}
Sep 5, 09:51:30 PM: 64f72c4c INFO [POST] /api/1/upload/test3 (SSR)
Sep 5, 09:51:30 PM: 64f72c4c Duration: 403.09 ms Memory Usage: 111 MB Init Duration: 532.42 ms

Note, I see neither "ERROR" nor "SUCCESS" in the log. More importantly, the file is almost never created the first time I click the button. When I click the upload button a second time (or sometimes a third or fourth), the file IS ultimately created.

I based this code on the answer to a thread on the old forum. And it DOES eventually work! But I don't understand why it only works intermittently, and why the callback seems to never run (or at least the console log never appears) on Netlify.

Am I doing something wrong here? I found this thread talking about Next.js functions failing on Netlify due to timeouts from excessive imports. So I made sure I was importing sparsely. 403ms seems reasonable for a cold start, right? Subsequent requests are much faster (e.g. 75ms), but are also apparently not running the upload callback. So while it's probably a good idea, that solution doesn't seem to have been the fix for my issue.

I realize the problem could be in Netlify/Next.js rather than in Cloudinary, but the route itself is being called reliably. It's the API call to Cloudinary that's having trouble. So I'm wondering if there's some idiosyncrasy of the upload_stream API which I'm unaware of. And, of course, if there's a better way to upload a JSON object to a file on Cloudinary, I'm all ears!

Thanks very much in advance.

nk9

I've worked out the problem. I didn't realize that all aspects of streams are asynchronous, even when they're chained together. So my code was behaving like this:

1. upload_stream() returned the writeable stream immediately.
2. The pipe was invoked asynchronously.
3. If it took a little too long, the Readable stream didn't get time to fully pass its contents to the upload_stream before the function ended, so the file wasn't uploaded.
4. In either case, the function was over by the time the cloudinary result callback was due to be called.

I worked this out after reading this GitHub issue thread. The solution was to wrap the upload in a Promise and use await to guarantee that the upload is complete before the API function ends.

import { v2 as cloudinary } from 'cloudinary'

// Based on https://github.com/cloudinary/cloudinary_npm/issues/130#issuecomment-865314280
function uploadStringAsync(str, album_name) {
    return new Promise((resolve, reject) => {
        cloudinary.uploader.upload_stream(
            {
                public_id: 'info.json',
                resource_type: 'raw',
                folder: `albums/${album_name}`
            },

            function onEnd(error, result) {
                if (error) {
                    console.log("ERROR:", error, result);
                    return reject(error)
                }

                console.log("Uploaded info.json:", str)
                resolve(result)
            }
        )
            .end(str)
    })
}

export default async function uploadInfoJSON(req, res) {
    if (req.method !== 'POST') {
        res.status(405).send({ message: 'Only POST requests allowed' })
        return
    }

    cloudinary.config({
        cloud_name: process.env.NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME,
        api_key: process.env.CLOUDINARY_KEY,
        api_secret: process.env.CLOUDINARY_SECRET,
        secure: true
    })

    const { slug } = req.query;
    const jsonString = JSON.stringify(req.body, null, 2)

    await uploadStringAsync(jsonString, slug)

    res.status(200).end()
};

DannyFromCloudinary

Hey @nk9

Thanks for getting in touch. I'm glad you were able to get to the bottom of it, and we really appreciate you following up once you had found the solution, as this will allow others who face a similar issue in the future to have a better idea of what the cause and resolution is.

If there's anything you need from us though, please don't hesitate to get in touch.

Kind regards,

-Danny