Common Content-Type and Character-Encoding API Bugs

Problem and expected outcome

A JSON-looking body may be parsed as text because the media type is wrong, while valid Unicode may be corrupted when bytes and charset assumptions differ. Form encoding adds a separate plus-versus-space rule that percent decoding alone does not provide.

Test beyond English ASCII and inspect headers and raw bytes when replacement characters appear.

Debugging sequence

• Capture Content-Type and Content-Encoding separately.
• Confirm the body bytes match the declared representation and compression.
• Test names and values containing accents, CJK characters, emoji, combining marks, plus signs, percent signs, and newlines.
• Compare application/json with form and multipart handling.
• Verify clients reject or clearly report unsupported media types and malformed byte sequences.

Synthetic example

POST /v1/messages HTTP/1.1
Content-Type: application/json; charset=utf-8

{"message":"Bună 👋 + 100%"}

QA checks and boundaries

• UTF-8 text round-trips without replacement characters.
• A literal plus remains plus in JSON and is only space-decoded in form mode.
• Compressed responses declare and apply Content-Encoding correctly.
• JSON endpoints do not return an HTML error document with a 2xx status.

Security and false assumptions

• Base64 does not repair an undefined character encoding.
• Do not double-decode percent sequences.
• MIME sniffing should not replace an explicit correct media type.

Inspect bytes before blaming JSON

Capture raw response bytes and headers when replacement characters or parse failures appear. Content-Type describes the media format, Content-Encoding describes transformations such as gzip, and transfer framing is another layer. Applying the wrong decompression or text decoder can corrupt an otherwise correct JSON document.

JSON exchanged over modern APIs should use Unicode and is typically UTF-8. A charset parameter that disagrees with actual bytes creates inconsistent client behavior. Confirm what the server emits rather than trusting a framework default.

Form and query behavior is different

application/x-www-form-urlencoded uses percent encoding with plus commonly representing space. JSON has no such plus rule: a plus inside a JSON string is a plus. Multipart bodies add boundaries and per-part headers. Reusing one decoder for all three formats causes silent data changes.

Test exact wire bodies containing a literal plus, %2B, %20, an ampersand, equals sign, emoji, combining sequence, and newline. Assert the value after each parsing layer.

Diagnostic checklist

• Final request and response Content-Type values after proxies.
• Content-Encoding and whether bytes were decompressed once.
• Hex or byte view around the first corrupted character.
• Declared versus actual charset.
• Parser selected by the client library.
• A Unicode-rich synthetic fixture that reproduces the failure.

References

• RFC 9110, Media Type
• WHATWG Encoding Standard
• RFC 8259, Character Encoding

FAQ

Related guides