Exercise: Photo Contest Uploads

Finish Snapshot Saturday's API, accepting one photo per entry with multer's size and type limits, answering 202 while a small job queue makes each thumbnail after the response, and streaming every entry as a CSV download with pipeline, with Supertest and node --test checking each part.

Freelancer · Node.js · Web Development · Updated

By

Snapshot Saturday, the town's photo contest, opens for entries next week. You'll finish its API: entrants send a photo with their name and its title, the API checks the photo's size and type and saves it under a name it makes, answers at once with 202 while a background job makes the thumbnail, and the judges download every entry as a CSV. Tests check every part as you go.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-node-photo-contest-uploads, also linked from the Exercise box beside the lesson. Clone it into your nodejs-course folder. In VS Code's terminal:

cd ~/nodejs-course
git clone https://github.com/ask-codey/exercise-node-photo-contest-uploads.git

Git makes a folder, exercise-node-photo-contest-uploads, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and install the packages:

npm install

Get the Exercise Code with Git walks through cloning step by step, if you'd like a reminder.

The folder holds:

  • app.js: the Express app, export const app = express(), for you to finish, with sendProblem(res, status, detail) from Error Handling and Consistent Error Responses ready to send every error.
  • queue.js: the job queue, addJob(work) and whenIdle(), for you to finish.
  • entries.js: the contest's three entries so far, kept in memory, with nextId() and resetEntries().
  • csv.js: csvHeader and toCsvLine(entry), which turn entries into lines of CSV text, commas and quotes taken care of.
  • thumbnails.js: makeThumbnail(photo), a stand-in for an image library that waits 100 milliseconds and answers with the thumbnail's name.
  • folders.js: folders.uploads, the folder photos are saved in. The tests point it at a temporary folder of their own.
  • server.js: makes the uploads folder and starts the API at http://localhost:3100.
  • fixtures/: three small files to upload, sunset.jpg, lighthouse.png and entry-rules.txt.
  • contest.http: the requests to try with REST Client, uploads included.
  • app.test.js and queue.test.js: the tests. You don't need to change them.

What to do

app.js answers every request with a 404 that says it isn't built yet, and both functions in queue.js throw an error that says the same. The comment at the top of app.js lists every request. Every error answer has the course's one error shape, { status, title, detail }, with the exact detail shown:

//   GET  /entries.csv   200, every entry as CSV, streamed with pipeline (not built up in one string first):
//                         Content-Type text/csv; charset=utf-8, Content-Disposition attachment; filename=entries.csv
//                         the body is csvHeader, then toCsvLine(entry) for each entry in entries, in order
//   GET  /entries/2     200, entry 2 (every entry's id works the same way)
//   GET  /entries/99    404, { "status": 404, "title": "Not Found", "detail": "There's no entry 99." }
//
//   POST /entries       a multipart/form-data form: the text fields name and title, and one photo in the field photo.
//                       202, the new entry: { id: nextId(), name, title, photo, thumbnail: null },
//                         also added to entries, with the header Location: /entries/<its id>.
//                         photo is the name the photo was saved under, in the folder folders.uploads:
//                         a new random UUID from node:crypto, then .jpg for image/jpeg or .png for image/png,
//                         never the file name the client sent.
//                       Then, after the answer, a job on the queue (addJob in queue.js) sets the entry's
//                         thumbnail to await makeThumbnail(photo).
//                       400, { "status": 400, "title": "Bad Request", "detail": "Only JPEG and PNG photos can enter." }
//                         for any other type of file
//                       413, { "status": 413, "title": "Payload Too Large", "detail": "A photo can be at most 5 MB." }
//                         for a photo over 5 MB (5 * 1024 * 1024 bytes; exactly 5 MB is fine)
//                       400, { "status": 400, "title": "Bad Request", "detail": "Attach one photo in the photo field." }
//                         with no file, or a file in any other field
//                       A refused upload leaves no file behind in the upload folder.

And the comment at the top of queue.js describes the queue:

//   addJob(work)  work is an async function. addJob puts it at the back of the queue and returns at once,
//                 without waiting for the job to finish.
//                 Jobs run one at a time, in the order they were added: a job starts only when the one before
//                 it has finished.
//                 A job that throws doesn't stop the queue: log it with
//                   console.error(`A background job failed: ${err.message}`)
//                 and go on to the next job.
//   whenIdle()    Returns a promise that resolves once every job added so far has finished (or failed).
//                 When no job is waiting or running, it resolves at once.
  1. Build the queue in queue.js: addJob first, then whenIdle. Run npm test and watch queue.test.js turn green.
  2. Answer GET /entries/:id with the entry, or the 404.
  3. Stream GET /entries.csv with pipeline, as a download named entries.csv.
  4. Accept a photo with multer: saved in folders.uploads under a name the server makes, with the size limit and the allowed types, answering 202 with the Location header.
  5. Queue the thumbnail after the answer, with addJob.
  6. Turn every refusal into its answer in an error-handling middleware, with sendProblem: 413 for a photo that's too big, 400 for the wrong type or the wrong field.

Check your work

Run the tests from the exercise folder:

npm test

No server needs to be running: the tests hand each request straight to app, and they save uploads in a temporary folder they make and remove themselves. At first all 19 tests fail. That's expected: each part you build turns its tests green. A failing test's message says what came back, for example POST /entries should answer 202, but it answered 404: {"message":"Not built yet."}

To try the API yourself, start it with npm start (or npm run dev, which restarts it each time you save), open contest.http, and click Send Request above each request. Your uploads land in the uploads folder, which .gitignore keeps out of Git. Stop the server with Ctrl+C.

Hints

  • Read the folder at the moment of each upload. destination can be a function, (req, file, cb) => cb(null, folders.uploads), so it sees the folder the tests choose. File Uploads with Limits shows diskStorage with a filename function in the same style.
  • The extension comes from your own list. An object such as { "image/jpeg": ".jpg", "image/png": ".png" } answers both questions: is this type allowed, and which extension does it get?
  • multer's errors have a code. A file over the limit is LIMIT_FILE_SIZE, and a file in a field other than photo is LIMIT_UNEXPECTED_FILE. File Uploads with Limits gives your own fileFilter error a status of 400 for the error-handling middleware to read.
  • The CSV is a list of lines. Readable.from([csvHeader, ...entries.map(toCsvLine)]) is a readable stream, and the response is the writable one. Streams: Big Files a Piece at a Time sends a download with res.attachment and pipeline.
  • The job changes the entry itself. Inside the job, entry.thumbnail = await makeThumbnail(entry.photo) updates the same object that's in entries, so GET /entries/4 sees it. Answer with res.status(202).location(...) as Work That Runs After the Response does.
  • whenIdle keeps promises waiting. Keep the resolve function of each promise it hands out in a list, new Promise((resolve) => waiters.push(resolve)), and call them all when runNext finds nothing left to do.

When you're done

Every test is green, and with npm start running, contest.http's requests get these answers (each request, then its answer's status line, the headers that matter, and the body). Your photo names are new UUIDs, so yours differ:

GET /entries.csv
HTTP/1.1 200 OK
Content-Type: text/csv; charset=utf-8
Content-Disposition: attachment; filename=entries.csv
id,name,title,photo,thumbnail
1,Ava,Harbor at dawn,3f2b7c1e-5d4a-4c8e-9b1f-2a6d8e0c4b71.jpg,thumb-3f2b7c1e-5d4a-4c8e-9b1f-2a6d8e0c4b71.jpg
2,Mateo,"Fog, over the bridge",8c1d4e2a-7b3f-4a9c-8e5d-1f0b6c2a9d34.png,thumb-8c1d4e2a-7b3f-4a9c-8e5d-1f0b6c2a9d34.png
3,Grace,"The ""lucky"" oak",b7e9a0d3-2c6f-4e1b-a8d4-5f3c9e7b0a12.jpg,thumb-b7e9a0d3-2c6f-4e1b-a8d4-5f3c9e7b0a12.jpg

POST /entries, Noor's sunset.jpg
HTTP/1.1 202 Accepted
Location: /entries/4
{"id":4,"name":"Noor","title":"Lighthouse in the rain","photo":"8e1b461a-6daa-408c-a58d-f5f7123a663c.jpg","thumbnail":null}

GET /entries/4, a moment later
HTTP/1.1 200 OK
{"id":4,"name":"Noor","title":"Lighthouse in the rain","photo":"8e1b461a-6daa-408c-a58d-f5f7123a663c.jpg","thumbnail":"thumb-8e1b461a-6daa-408c-a58d-f5f7123a663c.jpg"}

POST /entries, entry-rules.txt
HTTP/1.1 400 Bad Request
{"status":400,"title":"Bad Request","detail":"Only JPEG and PNG photos can enter."}

POST /entries, a 6 MB photo
HTTP/1.1 413 Payload Too Large
{"status":413,"title":"Payload Too Large","detail":"A photo can be at most 5 MB."}

The CSV puts Mateo's title in double quotes because of its comma, and writes each " in Grace's title twice: that's how a CSV keeps one value in one column.

Now look back at your work. If the server restarted between Noor's 202 and her thumbnail, what would GET /entries/4 show, and what would a job service such as BullMQ do differently?

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-node-photo-contest-uploads), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-node-photo-contest-uploads.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    

    Your test uploads stay behind: .gitignore keeps the uploads folder out of git add ..

  3. Paste your repo's address, such as https://github.com/your-name/exercise-node-photo-contest-uploads, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

Uploads with limits, a quick 202 with the slow part queued, and a download that streams: that's the shape of most real file features, from profile pictures to monthly reports, and you've now built each piece and tested it.

Key terms

Related