Python Write to File Line by Line

Introduction

Writing data to files is a core task for any Python developer. Whether you're logging events, generating reports, or storing user input, writing one line at a time gives you control and readability. In this article, we'll walk through multiple approaches to write to files line by line in Python.

Understanding how to write incrementally can help you avoid memory spikes, handle large datasets, and make your logs more organized. Ready to see how small changes in your code can make file I/O smoother and more reliable?

By mastering these simple techniques, you'll write cleaner, more efficient code and prevent common pitfalls like file corruption or unexpected overwrites. Let's dive in!

Why Write Line by Line

Writing line by line lets you:

• Stream large datasets without loading everything in memory
• Interleave processing and output for real-time logging
• Append new entries without rewriting the entire file

Tip: When dealing with huge files, writing one line at a time keeps memory usage low and avoids MemoryError.

Python’s built-in file object makes this straightforward. You’ll see two main patterns: using a loop with write() and leveraging writelines() for lists.

Using a Loop with write()

The most explicit way is to open a file and call write() for each line:

lines = [
    "First line of output\n",
    "Second line of output\n",
    "Third line of output\n"
]

with open('output.txt', 'w', encoding='utf-8') as f:
    for line in lines:
        f.write(line)

Key points:

• The with statement ensures the file closes automatically, even on errors.
• Always include a newline \n at the end of each line.
• Choose the right mode: 'w' for overwrite, 'a' to append.

Append Mode

To add new lines without erasing existing data:

with open('output.txt', 'a', encoding='utf-8') as f:
    f.write("Another line added later\n")

Writing from an Iterable with writelines()

If you already have a list or generator of lines, writelines() is concise:

log_entries = (f"Entry {i}: data processed\n" for i in range(1000))

with open('log.txt', 'w') as logfile:
    logfile.writelines(log_entries)

Note:

• writelines() does not add newlines automatically. Each element must end with \n.
• It writes all lines in one go, which can be faster but uses more memory.

Tip: If you need to save a Python list, see our guide on saving a Python list to a file.

Flushing and Buffering

Python buffers writes for performance. In most cases, you don’t need to worry about it. But for real-time logs or progress updates:

with open('progress.txt', 'w') as f:
    for i in range(5):
        f.write(f"Step {i}\n")
        f.flush()  # Force write to disk
        time.sleep(1)

Warning: Excessive flushing can slow down your program.

Handling File Encodings

Different platforms use different default encodings. To avoid surprises:

with open('data.txt', 'w', encoding='utf-8', errors='replace') as f:
    f.write("Some unicode text: café, 北京\n")

• encoding='utf-8' ensures consistent behavior across systems
• errors='replace' or 'ignore' handles unexpected byte sequences

Serializing Complex Data

When you need structured formats like JSON, write line by line as serialized strings:

import json
records = [
    {'id': 1, 'status': 'ok'},
    {'id': 2, 'status': 'error'}
]

with open('records.jsonl', 'w') as f:
    for rec in records:
        line = json.dumps(rec)
        f.write(line + "\n")

Pro tip: JSON Lines (.jsonl) files are great for streaming large logs.

For more advanced JSON file writing, check out our writing JSON to files in Python guide.

Common Pitfalls and Best Practices

• Missing newline: Forgetting \n results in mashed lines.
• Wrong mode: Using 'w' instead of 'a' overwrites existing data.
• No exception handling: Wrap critical writes in try/except if failure is possible.
• Hard-coded paths: Use os.path or pathlib to build file paths.

Best Practices:

1. Always use with open(...).
2. Specify encoding explicitly.
3. Choose the smallest buffer you need.
4. Flush sparingly.

Conclusion

Line-by-line writing in Python is simple yet powerful. You can stream large data, create real-time logs, or serialize complex records without loading everything into memory. By using write(), writelines(), and careful buffering, your applications will run efficiently and predictably. Remember to handle encodings explicitly and choose the correct file mode to avoid data loss.

Next time you need to record output or logs, lean on these patterns to keep your code clean and your files intact. Happy coding!