DuckDB for SQL Server DBAs: Analyze CSV and Parquet Files with SQL in Minutes

DBA PARK may earn a commission from purchases through links in this article, at no extra cost to you.

DuckDB has become one of the most discussed database systems in data engineering. It passed 40,000 GitHub stars in August 2026, and the project reports more than 50 million monthly downloads from PyPI. More importantly, it solves a problem that many DBAs and developers know well: how can I analyze a large CSV or Parquet file without first building a server, database, table, and import process?

With DuckDB, the answer can be a single SQL statement.

This article explains DuckDB from a SQL Server DBA’s perspective. We will install it, create a small analytical workload, query Parquet directly, and clarify where DuckDB fits—and where SQL Server remains the better choice.

Version note: The examples in this article use DuckDB 1.5.5, the current stable release at the time of writing. DuckDB 2.0 has been announced for fall 2026, but preview features are discussed separately and are not required for the hands-on examples.

What Is DuckDB?

DuckDB is an open-source relational database designed primarily for analytical workloads, also known as OLAP workloads. It supports SQL, transactions, window functions, joins, common table expressions, and many features familiar to relational database users.

The unusual part is its deployment model.

Traditional SQL Server usage normally looks like this:

Application or SSMS → network connection → SQL Server service → database files

DuckDB normally runs inside the process that uses it:

CLI, Python, R, or application → DuckDB library → local file or in-memory database

There is no database service to configure for the normal embedded use case. The DuckDB CLI is a single executable, and a persistent database can be stored in one file.

This makes DuckDB feel similar to SQLite operationally, but its query engine is designed for analytics rather than small transactional application workloads. DuckDB uses vectorized execution and a column-oriented engine to efficiently scan and aggregate large data sets.

Why Is DuckDB Becoming Popular?

DuckDB removes several steps between receiving data and querying it.

Suppose someone sends you a 5 GB CSV file and asks for totals by month. With SQL Server, a typical workflow is:

1. Create or select an instance.
2. Create a database and table.
3. Define the column data types.
4. Import the file with BULK INSERT, BCP, SSIS, or another tool.
5. Troubleshoot delimiters, encoding, and conversion errors.
6. Run the query.

DuckDB can automatically detect CSV settings and data types, then query the file directly:

SELECT
    date_trunc('month', order_date) AS order_month,
    sum(amount) AS total_amount
FROM read_csv('sales.csv')
GROUP BY ALL
ORDER BY order_month;

DuckDB can also query Parquet files directly. Because Parquet stores column metadata and statistics, DuckDB can skip unnecessary columns and row groups. This is especially useful for ad hoc analysis, ETL validation, log analysis, data science, and local processing of files from object storage.

DuckDB vs. SQL Server

DuckDB is not a smaller replacement for SQL Server. The two products optimize for different workloads.

The easiest mental model is:

SQL Server is a database server. DuckDB is an analytical SQL engine that you can bring to the data.

Install DuckDB on Windows

The simplest Windows installation is through WinGet:

winget install DuckDB.cli

Alternatively, download the Windows CLI ZIP from the official DuckDB installation page, extract duckdb.exe, and run it from PowerShell.

Verify the installation:

duckdb --version

Start an in-memory session:

duckdb

Or create a persistent database file:

duckdb dba_lab.duckdb

Unlike attaching an MDF file, opening dba_lab.duckdb does not require a running database service. The CLI process opens the database directly.

Hands-On: Create One Million Sales Rows

We can generate test data entirely in SQL. Start DuckDB with duckdb dba_lab.duckdb, then execute the following statement:

CREATE TABLE sales AS
SELECT
    order_id,
    DATE '2025-01-01'
        + CAST(order_id % 365 AS INTEGER) AS order_date,
    order_id % 10000 AS customer_id,
    CASE order_id % 4
        WHEN 0 THEN 'North'
        WHEN 1 THEN 'South'
        WHEN 2 THEN 'East'
        ELSE 'West'
    END AS region,
    round(((order_id * 17) % 50000) / 100.0, 2) AS amount
FROM range(1, 1000001) AS t(order_id);

Confirm the row count:

SELECT count(*) AS row_count
FROM sales;

Expected result:

1000000

Now aggregate the rows by region:

SELECT
    region,
    count(*) AS order_count,
    round(sum(amount), 2) AS total_amount,
    round(avg(amount), 2) AS average_amount
FROM sales
GROUP BY region
ORDER BY total_amount DESC;

The syntax should be comfortable for a SQL Server user. Some functions and data types differ, but joins, grouping, window functions, CTEs, and transactions follow familiar relational concepts.

Export the Table to Parquet

Export the table with one statement:

COPY sales
TO 'sales.parquet'
(FORMAT PARQUET, COMPRESSION ZSTD);

You can now query the Parquet file without importing it into another table:

SELECT
    region,
    sum(amount) AS total_amount
FROM read_parquet('sales.parquet')
WHERE order_date >= DATE '2025-07-01'
GROUP BY region
ORDER BY total_amount DESC;

DuckDB only needs the columns referenced by the query. It can also use Parquet metadata to avoid reading row groups that cannot satisfy a filter. In SQL Server terms, this is closer to querying an external columnar data source than scanning a conventional heap.

You can inspect the execution plan with:

EXPLAIN ANALYZE
SELECT
    region,
    sum(amount) AS total_amount
FROM read_parquet('sales.parquet')
WHERE order_date >= DATE '2025-07-01'
GROUP BY region;

Look for the TABLE_SCAN section. It shows the projected columns, filters, processed rows, and timing.

Query CSV Without Creating a Table

First, create a CSV file from the sample table:

COPY sales
TO 'sales.csv'
(HEADER, DELIMITER ',');

Then query the file directly:

SELECT
    region,
    count(*) AS order_count
FROM read_csv('sales.csv')
GROUP BY region
ORDER BY region;

DuckDB’s CSV sniffer attempts to detect the delimiter, header, quoting rules, and column types. When automatic detection is not correct, specify the options explicitly:

SELECT *
FROM read_csv(
    'sales.csv',
    delim = ',',
    header = true,
    columns = {
        'order_id': 'BIGINT',
        'order_date': 'DATE',
        'customer_id': 'BIGINT',
        'region': 'VARCHAR',
        'amount': 'DOUBLE'
    }
);

For repeatable production pipelines, explicit types are safer than relying indefinitely on inference. A future file might contain values that cause a column to be inferred differently.

A Useful DBA Scenario: Analyze Exported Logs

DuckDB is particularly useful when troubleshooting data is spread across multiple files. For example, if every server exports the same CSV structure, a wildcard can read them as one data set:

SELECT
    filename,
    error_number,
    count(*) AS occurrences
FROM read_csv(
    'logs/*.csv',
    filename = true,
    union_by_name = true
)
WHERE severity >= 16
GROUP BY filename, error_number
ORDER BY occurrences DESC;

This avoids creating a temporary SQL Server database only to combine diagnostic exports. The filename option preserves the source file, while union_by_name helps when columns appear in a different order or when the file schemas evolve.

DuckDB will not parse every proprietary diagnostic format automatically. SQL Server error logs, Extended Events, network traces, and dump files may still need preprocessing or specialized tools. It is most useful after the relevant records have been exported to CSV, JSON, or Parquet.

Can DuckDB Query SQL Server?

A community-maintained mssql extension can attach SQL Server, Azure SQL, or Microsoft Fabric as a DuckDB catalog over TDS:

INSTALL mssql FROM community;
LOAD mssql;

ATTACH 'Server=localhost,1433;Database=AdventureWorks;User Id=report_user;Password=your_password'
AS sqlserver_db (TYPE mssql);

SELECT *
FROM sqlserver_db.Sales.SalesOrderHeader
LIMIT 10;

This creates interesting possibilities, such as joining a SQL Server table to a local Parquet file. However, this is a community extension rather than a core DuckDB component. Before using it with production systems:

• Test compatibility and TLS requirements.
• Use a least-privileged, read-only account.
• Do not place plaintext credentials in scripts or shell history.
• Validate which predicates are pushed to SQL Server.
• Test performance with realistic row counts.

For critical production integrations, use an approved and supported data movement method unless your organization has evaluated the extension.

Important Limitations

DuckDB’s simplicity can be misleading if it is evaluated as a direct SQL Server replacement.

1. It Is Optimized for Analytics

DuckDB performs well when scanning, joining, and aggregating many rows. A workload containing thousands of tiny transactions from many application sessions is not its primary design goal.

2. Embedded Concurrency Is Different

In the traditional embedded model, one process can read and write a DuckDB database file. Multiple processes can open it concurrently in read-only mode. This is very different from a SQL Server instance serving many independent clients.

The Quack remote protocol is expanding multi-process and client/server options, but it is still beta in DuckDB 1.5.x. Do not design a critical production service around preview behavior without testing and an upgrade plan.

3. High Availability Is Not an Availability Group

A local DuckDB file does not provide the operational platform that SQL Server DBAs expect from Availability Groups, Failover Cluster Instances, SQL Agent, centralized access control, and mature backup tooling.

4. Direct File Access Requires Discipline

A single-file database is convenient, but copying or modifying a file while it is actively used can cause operational problems. Treat the database file as managed data, not as an ordinary document.

What Is Coming in DuckDB 2.0?

DuckDB 2.0 is planned for fall 2026. The announced headline is DuckDB as a server.

The Quack protocol and the new CONNECT statement are intended to allow one DuckDB process to serve a database remotely. The preview also includes triggers, a first-class VARIANT type for semi-structured data, asynchronous I/O, a new SQL parser, and a new default storage format.

For SQL Server professionals, this is significant because DuckDB is beginning to cover scenarios beyond single-user, in-process analytics. However, v2.0 is not generally available at the time of writing. Treat the feature list and syntax as preview information until the final release and documentation are published.

When Should a SQL Server DBA Use DuckDB?

DuckDB is a strong choice when you need to:

• Inspect a large CSV, JSON, or Parquet file quickly.
• Aggregate exported monitoring or application data.
• Validate data before or after an ETL process.
• Build a local analytical prototype.
• Use SQL inside Python, R, or a notebook.
• Convert data between CSV and Parquet.
• Join local files without loading them into a permanent server.

SQL Server remains the stronger choice when you need to:

• Support a shared transactional application.
• Handle many concurrent client connections and writes.
• Implement mature HA/DR and point-in-time recovery procedures.
• Use SQL Agent, enterprise auditing, centralized security, or established operations tooling.
• Obtain Microsoft product support for the complete database platform.

Conclusion

DuckDB is useful not because it replaces SQL Server, but because it removes database-server work from tasks that do not need a database server.

For a SQL Server DBA, the learning curve is small: download one executable, open a file or an in-memory database, and use familiar SQL. The biggest conceptual change is that CSV and Parquet files can be queried where they are instead of being imported first.

If your next troubleshooting or reporting task begins with “someone sent me several large files,” DuckDB is worth trying before creating another staging database.

Continue learning: To build a larger analytics project from these examples, browse Udemy and search for DuckDB courses with exercises on analytical SQL, Python integration, and file-based workflows. To strengthen the SQL querying and relational database fundamentals behind your work, explore the interactive courses on DataCamp.

References

• Why DuckDB
• DuckDB installation
• DuckDB CSV auto detection
• DuckDB concurrency
• DuckDB 2.0 preview
• DuckDB reaches 40,000 GitHub stars
• DuckDB community MSSQL extension