Encoding test files: UTF-16, BOM and Latin-1
Encoding bugs are the ones users report as "the file is fine, your system is broken", and they are usually right that the file is fine. It is simply not the encoding your reader assumed.
Any importer that accepts files from outside your organization will eventually receive all of these, because the tools people export from disagree about what a text file is.
The byte-order mark: three invisible bytes
A UTF-8 byte-order mark is the three bytes EF BB BF at the start of a file. It carries no visible character, editors hide it, and it silently breaks anything that requires a specific first byte.
It breaks more than you would expect. JavaScript's JSON.parse throws on a leading BOM. Python's json module rejects it unless the file is read as utf-8-sig. A CSV's first column header becomes something that no longer matches your mapping, so the import fails on a column that is visibly present. A shell script's shebang is no longer the first thing in the file, so the interpreter is never selected.
Windows tools add it routinely, which is why a spreadsheet exported on one machine imports cleanly and the same export from a colleague does not.
UTF-16: every byte doubled, and a zero between each one
UTF-16 stores each character in a two-byte unit, so ASCII text becomes twice as long with a zero byte after every character. Read as UTF-8, that is not slightly wrong. It is unreadable, and often looks like a binary file to code that sniffs content.
Two consequences are worth testing for directly. A UTF-16 file always has an even byte count, because every character occupies two bytes; an odd length means truncation. And byte order matters: little-endian starts FF FE, big-endian starts FE FF, and a reader that assumes the wrong one produces text made entirely of the wrong characters rather than failing outright.
Windows calls UTF-16 LE simply "Unicode" in save dialogs, which is how people produce these files without realising. It is also the encoding Windows uses for registry export files.
Generating these deliberately
On this site encoding is a control of its own rather than a content choice, offering UTF-8, UTF-8 with BOM, UTF-16 LE and UTF-16 BE for the plain-text formats. The file is generated as ASCII, re-encoded, and the body is shortened first so the finished file still lands on the exact size you asked for.
It applies only to genuinely text-based formats. A DOCX, XLSX or PDF is a container that happens to hold text, and rewriting its bytes as UTF-16 would not produce a UTF-16 document. It would produce a broken container, which is a corruption test dressed up as an encoding test.
One honest limitation: a UTF-16 file cannot have an odd byte count, so an odd custom size is reduced by one byte. Every preset size is even, so this only appears if you type an odd number yourself.
What to assert
- A BOM is either handled or rejected explicitly, never allowed to become part of the first field's value.
- The encoding is detected from the file, or required from the user, rather than assumed.
- Character counts and byte counts are not used interchangeably in validation messages or column limits.
- Non-ASCII content survives a round trip: import, store, export, and compare against the original.
- A truncated multi-byte character at the end of a file fails cleanly rather than producing a replacement character in stored data.
Common questions
What is a BOM and why does it break my import?
A byte-order mark is a few invisible bytes at the start of a file: EF BB BF for UTF-8. Editors hide it, but anything requiring a specific first byte fails: JSON.parse throws, a CSV's first header no longer matches, and a shebang stops working.
Why is my UTF-16 file twice the size?
UTF-16 stores every character in two bytes, so ASCII text doubles in length with a zero byte after each character. This is also why a UTF-16 file always has an even byte count.
Which encodings can I generate here?
UTF-8, UTF-8 with BOM, UTF-16 LE and UTF-16 BE, for the plain-text formats. Container formats such as DOCX and PDF are excluded on purpose, because re-encoding their bytes would corrupt them rather than change their encoding.
Formats mentioned here: CSV test files · JSON test files · TXT test files · XML test files · SQL test files
More guides
How to create a very large test file · How to create an invalid JSON file for testing · How to make a corrupted file for testing · How to test file upload validation · Filename test cases that break real systems