Controlling maximum length line via truncation
The sql_formatter will try to truncate too long lines in the SELECT clause for either
- Function with many arguments
inwith many elements
The default maximum line length is 82 after line stripping.
You can control the maximum length line using e.g.
sql-formatter sql_file.sql --max-line-length=50
pre-commit is a nice development tool to automatize the binding of pre-commit hooks. After installation and configuration pre-commit will run your hooks before you commit any change.
To add sql-formatter as a hook to your pre-commit configuration to format your SQL files before commit, just add the following lines to your .pre-commit-config.yaml:
repos:
- repo: https://github.com/PabloRMira/sql_formatter
rev: master
hooks:
- id: sql_formatter
If you want to install sql-formatter locally and use that instead of using pre-commit's default environment, set repo: local in your .pre-commit-config.yaml file:
repos:
- repo: local
hooks:
- id: sql_formatter
name: SQL formatter
language: system
entry: sql-formatter
files: \.sql$
or
repos:
- repo: local
hooks:
- id: sql_formatter
name: SQL formatter
language: system
entry: sql-formatter --max-line-length=50
files: \.sql$
for a custom maximum line length truncation of e.g. 50
To exemplify the formatting let's say you have a SQL query like this
example_sql = """
create or replace table mytable as -- mytable example
seLecT a.asdf, b.qwer, -- some comment here
c.asdf, -- some comment there
b.asdf2 frOm table1 as a leFt join
table2 as b -- and here a comment
on a.asdf = b.asdf -- join this way
inner join table3 as c
on a.asdf=c.asdf
whEre a.asdf= 1 -- comment this
anD b.qwer =2 and a.asdf<=1 --comment that
or b.qwer>=5
groUp by a.asdf
"""
Then you can use this package to format it so that it is better readable
from sql_formatter.core import format_sql
print(format_sql(example_sql))
It can even deal with subqueries and it will correct my favourite simple careless mistake (comma at the end of SELECT statement before of FROM) for you on the flow :-)
print(format_sql("""
select asdf, cast(qwer as numeric), -- some comment
qwer1
from
(select asdf, qwer, from table1 where asdf = 1) as a
left
join (select asdf, qwer2 from table2 where qwer2 = 1) as b
on a.asdf = b.asdf
where qwer1 >= 0
"""))
The formatter is also robust against nested subqueries
print(format_sql("""
select field1, field2 from (select field1,
field2 from (select field1, field2,
field3 from table1 where a=1 and b>=100))
"""))
If you do not want to get some query formatted in your SQL file then you can use the marker /*skip-formatter*/ in your query to disable formatting for just the corresponding query
from sql_formatter.format_file import format_sql_commands
print(format_sql_commands(
"""
use database my_database;
-- My first view --
create or repLace view my_view as
select asdf, qwer from table1
where asdf <= 10;
/*skip-formatter*/
create oR rePlace tabLe my_table as
select asdf
From my_view;
"""
))
A note of caution
For the SQL-formatter to work properly you should meticulously end each of your SQL statements with semicolon (;)
However, we have equiped the sql-formatter with some basic validations:
- Forgotten semicolon validation: The validator will check if the
CREATEkeyword appears more than twice, indicating the user that he / she may have forgotten a semicolon - Unbalanced parenthesis: The validator will check if there are unbalanced parenthesis in the query
- Unbalanced
case when ... end: The validator will check if there arecase whenstatements withoutendor vice versa
What sql_formatter does not do
This package is just a SQL formatter and therefore
- cannot parse your SQL queries into e.g. dictionaries
- cannot validate your SQL queries to be valid for the corresponding database system / provider
Up to now it only formats queries of the form
CREATE TABLE / VIEW ...SELECT ...
Every other SQL commands will remain unformatted, e.g. INSERT INTO ...
Formatting Logic
The main goal of the sql_formatter is to enhance readability and quick understanding of SQL queries via proper formatting. We use indentation and lowercasing / uppercasing as means to arrange statements / clauses and parameters into context. By programmatically standardizing the way to write SQL queries we help the user understand its queries faster.
As a by-product of using the sql_formatter, developer teams can focus on the query logic itself and save time by not incurring into styling decisions, this then begin accomplished by the sql_formatter. This is similar to the goal accomplished by the black package for the Python language, which was also an inspiration for the development of this package for SQL.
We can summarize the main steps of the formatter as follows:
- Each query is separated from above by two newlines.
- Everything but main statements* / clauses is lowercased
* Main statements:
- CREATE ... TABLE / VIEW table_name AS
- SELECT (DISTINCT)
- FROM
- (LEFT / INNER / RIGHT / OUTER) JOIN
- UNION
- ON
- WHERE
- GROUP BY
- ORDER BY
- OVER
- PARTITION BY
- Indentation is used to put parameters into context. Here an easy example:
SELECT field1,
case when field2 > 1 and
field2 <= 10 and
field1 = 'a' then 1
else 0 end as case_field,
...
FROM table1
WHERE field1 = 1
and field2 <= 2
or field3 = 5
ORDER BY field1;
This is a very nice, easy example but things can become more complicated if comments come into play
- Subqueries are also properly indented, e.g.
SELECT a.field1,
a.field2,
b.field3
FROM (SELECT field1,
field2
FROM table1
WHERE field1 = 1) as a
LEFT JOIN (SELECT field1,
field3
FROM table2) as b
ON a.field1 = b.field1;
- Everything not being a query of the form
CREATE ... TABLE / VIEWorSELECT ...is left unchanged
We version our package via semantic versioning, i.e.,
- We use three digits separated by points x1.x2.x3, e.g. 0.5.1
- We increase x1 (the major version) if we introduce breaking changes
- Exception: Versions with 0 at the beginning (e.g. 0.5.1) mean that the package is not stable yet and therefore every new feature could be a breaking change
- We increase x2 (the minor version) if we introduce a new feature
- We increase x3 (the patch version) if we fix a bug
New documentation, refactoring / maintenance of code and admin tasks do not change the versions.
You can follow the changes introduced by each version in our CHANGELOG
See CONTRIBUTING
Thank you very much to Jeremy Howard and all the nbdev team for enabling the fast and delightful development of this library via the nbdev framework.
For more details on nbdev, see its official tutorial
Thank you very much for the developers of the black package, which was also an inspiration for the development of this package