replace all submodule documentation with flags documentation

6 files changed, 91 insertions, 44 deletions

diff --git a/README.md b/README.md
index e8f72b6d..8e4e42d9 100644
--- a/README.md
+++ b/README.md
@@ -36,12 +36,14 @@ We provide packaging for various distributions, but here's a quick method to
 build from source.
 
 ```
-git clone --recurse-submodules --shallow-submodules -j8 https://github.com/helix-editor/helix
+git clone https://github.com/helix-editor/helix
 cd helix
 cargo install --path helix-term
+hx --fetch-grammars
+hx --build-grammars
 ```
 
-This will install the `hx` binary to `$HOME/.cargo/bin`.
+This will install the `hx` binary to `$HOME/.cargo/bin` and build tree-sitter grammars.
 
 Helix also needs its runtime files so make sure to copy/symlink the `runtime/` directory into the
 config directory (for example `~/.config/helix/runtime` on Linux/macOS, or `%AppData%/helix/runtime` on Windows).
@@ -56,6 +58,7 @@ that sets the variable to the install dir.
 [![Packaging status](https://repology.org/badge/vertical-allrepos/helix.svg)](https://repology.org/project/helix/versions)
 
 ## MacOS
+
 Helix can be installed on MacOS through homebrew via:
 
 ```
diff --git a/book/src/configuration.md b/book/src/configuration.md
index 2e4cab8a..2b29379e 100644
--- a/book/src/configuration.md
+++ b/book/src/configuration.md
@@ -131,6 +131,7 @@ Search specific options.
 ## LSP
 
 To display all language server messages in the status line add the following to your `config.toml`:
+
 ```toml
 [lsp]
 display-messages = true
diff --git a/book/src/guides/adding_languages.md b/book/src/guides/adding_languages.md
index 5844a48e..51256288 100644
--- a/book/src/guides/adding_languages.md
+++ b/book/src/guides/adding_languages.md
@@ -1,45 +1,68 @@
 # Adding languages
 
-## Submodules
+## Language configuration
 
-To add a new language, you should first add a tree-sitter submodule. To do this,
-you can run the command
-```sh
-git submodule add -f <repository> helix-syntax/languages/tree-sitter-<name>
-```
-For example, to add tree-sitter-ocaml you would run
-```sh
-git submodule add -f https://github.com/tree-sitter/tree-sitter-ocaml helix-syntax/languages/tree-sitter-ocaml
+To add a new language, you need to add a `language` entry to the
+[`languages.toml`][languages.toml] found in the root of the repository;
+this `languages.toml` file is included at compilation time, and is
+distinct from the `languages.toml` file in the user's [configuration
+directory](../configuration.md).
+
+```toml
+[[language]]
+name = "mylang"
+scope = "scope.mylang"
+injection-regex = "^mylang$"
+file-types = ["mylang", "myl"]
+comment-token = "#"
+indent = { tab-width = 2, unit = "  " }
 ```
-Make sure the submodule is shallow by doing
-```sh
-git config -f .gitmodules submodule.helix-syntax/languages/tree-sitter-<name>.shallow true
+
+These are the available keys and descriptions for the file.
+
+| Key                   | Description                                                   |
+| ----                  | -----------                                                   |
+| `name`                | The name of the language                                      |
+| `scope`               | A string like `source.js` that identifies the language. Currently, we strive to match the scope names used by popular TextMate grammars and by the Linguist library. Usually `source.<name>` or `text.<name>` in case of markup languages |
+| `injection-regex`     | regex pattern that will be tested against a language name in order to determine whether this language should be used for a potential [language injection][treesitter-language-injection] site. |
+| `file-types`          | The filetypes of the language, for example `["yml", "yaml"]`. Extensions and full file names are supported.  |
+| `shebangs`            | The interpreters from the shebang line, for example `["sh", "bash"]` |
+| `roots`               | A set of marker files to look for when trying to find the workspace root. For example `Cargo.lock`, `yarn.lock` |
+| `auto-format`         | Whether to autoformat this language when saving               |
+| `diagnostic-severity` | Minimal severity of diagnostic for it to be displayed. (Allowed values: `Error`, `Warning`, `Info`, `Hint`) |
+| `comment-token`       | The token to use as a comment-token                           |
+| `indent`              | The indent to use. Has sub keys `tab-width` and `unit`        |
+| `config`              | Language server configuration                                 |
+| `grammar`             | The tree-sitter grammar to use (defaults to the value of `name`) |
+
+## Grammar configuration
+
+If a tree-sitter grammar is available for the language, add a new `grammar`
+entry to `languages.toml`.
+
+```toml
+[[grammar]]
+name = "mylang"
+source = { git = "https://github.com/example/mylang", rev = "a250c4582510ff34767ec3b7dcdd3c24e8c8aa68" }
 ```
 
-or you can manually add `shallow = true` to `.gitmodules`.
+Grammar configuration takes these keys:
 
-## languages.toml
+| Key      | Description                                                              |
+| ---      | -----------                                                              |
+| `name`   | The name of the tree-sitter grammar                                      |
+| `path`   | A path within the grammar directory which should be built. Some grammar repositories host multiple grammars (for example `tree-sitter-typescript` and `tree-sitter-ocaml`) in subdirectories. This key is used to point `hx --build-grammars` to the correct path for compilation. When ommitted, the root of the grammar directory is used |
+| `source` | The method of fetching the grammar - a table with a schema defined below |
 
-Next, you need to add the language to the [`languages.toml`][languages.toml] found in the root of
-the repository; this `languages.toml` file is included at compilation time, and
-is distinct from the `language.toml` file in the user's [configuration
-directory](../configuration.md).
+Where `source` is a table with either these keys when using a grammar from a
+git repository:
 
-These are the available keys and descriptions for the file.
+| Key   | Description                                               |
+| ---   | -----------                                               |
+| `git` | A git remote URL from which the grammar should be cloned  |
+| `rev` | The revision (commit hash or tag) which should be fetched |
 
-| Key                 | Description                                                   |
-| ----                | -----------                                                   |
-| name                | The name of the language                                      |
-| scope               | A string like `source.js` that identifies the language. Currently, we strive to match the scope names used by popular TextMate grammars and by the Linguist library. Usually `source.<name>` or `text.<name>` in case of markup languages |
-| injection-regex     | regex pattern that will be tested against a language name in order to determine whether this language should be used for a potential [language injection][treesitter-language-injection] site. |
-| file-types          | The filetypes of the language, for example `["yml", "yaml"]`  |
-| shebangs            | The interpreters from the shebang line, for example `["sh", "bash"]` |
-| roots               | A set of marker files to look for when trying to find the workspace root. For example `Cargo.lock`, `yarn.lock` |
-| auto-format         | Whether to autoformat this language when saving               |
-| diagnostic-severity | Minimal severity of diagnostic for it to be displayed. (Allowed values: `Error`, `Warning`, `Info`, `Hint`) |
-| comment-token       | The token to use as a comment-token                           |
-| indent              | The indent to use. Has sub keys `tab-width` and `unit`        |
-| config              | Language server configuration                                 |
+Or a `path` key with an absolute path to a locally available grammar directory.
 
 ## Queries
 
@@ -51,18 +74,14 @@ gives more info on how to write queries.
 
 > NOTE: When evaluating queries, the first matching query takes
 precedence, which is different from other editors like neovim where
-the last matching query supercedes the ones before it. See
+the last matching query supersedes the ones before it. See
 [this issue][neovim-query-precedence] for an example.
 
 ## Common Issues
 
-- If you get errors when building after switching branches, you may have to remove or update tree-sitter submodules. You can update submodules by running
-    ```sh
-    git submodule sync; git submodule update --init
-    ```
-- Make sure to not use the `--remote` flag. To remove submodules look inside the `.gitmodules` and remove directories that are not present inside of it.
+- If you get errors when running after switching branches, you may have to update the tree-sitter grammars. Run `hx --fetch-grammars` to fetch the grammars and `hx --build-grammars` to build any out-of-date grammars.
 
-- If a parser is segfaulting or you want to remove the parser, make sure to remove the submodule *and* the compiled parser in `runtime/grammar/<name>.so`
+- If a parser is segfaulting or you want to remove the parser, make sure to remove the compiled parser in `runtime/grammar/<name>.so`
 
 - The indents query is `indents.toml`, *not* `indents.scm`. See [this](https://github.com/helix-editor/helix/issues/114) issue for more information.
 
diff --git a/book/src/install.md b/book/src/install.md
index 8b7d8fa2..db42509c 100644
--- a/book/src/install.md
+++ b/book/src/install.md
@@ -44,7 +44,7 @@ sudo dnf install helix
 ## Build from source
 
 ```
-git clone --recurse-submodules --shallow-submodules -j8 https://github.com/helix-editor/helix
+git clone https://github.com/helix-editor/helix
 cd helix
 cargo install --path helix-term
 ```
@@ -54,3 +54,9 @@ This will install the `hx` binary to `$HOME/.cargo/bin`.
 Helix also needs it's runtime files so make sure to copy/symlink the `runtime/` directory into the
 config directory (for example `~/.config/helix/runtime` on Linux/macOS). This location can be overriden
 via the `HELIX_RUNTIME` environment variable.
+
+## Building tree-sitter grammars
+
+Tree-sitter grammars must be fetched and compiled if not pre-packaged.
+Fetch grammars with `hx --fetch-grammars` (requires `git`) and compile them
+with `hx --build-grammars` (requires a C compiler).
diff --git a/book/src/languages.md b/book/src/languages.md
index 4c4dc326..c5a7708f 100644
--- a/book/src/languages.md
+++ b/book/src/languages.md
@@ -4,10 +4,28 @@ Language-specific settings and settings for particular language servers can be c
 
 Changes made to the `languages.toml` file in a user's [configuration directory](./configuration.md) are merged with helix's defaults on start-up, such that a user's settings will take precedence over defaults in the event of a collision. For example, the default `languages.toml` sets rust's `auto-format` to `true`. If a user wants to disable auto-format, they can change the `languages.toml` in their [configuration directory](./configuration.md) to make the rust entry read like the example below; the new key/value pair `auto-format = false` will override the default when the two sets of settings are merged on start-up:
 
-```
+```toml
 # in <config_dir>/helix/languages.toml
 
 [[language]]
 name = "rust"
 auto-format = false
 ```
+
+## Tree-sitter grammars
+
+Tree-sitter grammars can also be configured in `languages.toml`:
+
+```toml
+# in <config_dir>/helix/languages.toml
+
+[[grammar]]
+name = "rust"
+source = { git = "https://github.com/tree-sitter/tree-sitter-rust", rev = "a250c4582510ff34767ec3b7dcdd3c24e8c8aa68" }
+
+[[grammar]]
+name = "c"
+source = { path = "/path/to/tree-sitter-c" }
+```
+
+If a user has a `languages.toml`, only the grammars in that `languages.toml` are evaluated when running `hx --fetch-grammars` and `hx --build-grammars`. Otherwise the [default `languages.toml`](https://github.com/helix-editor/helix/blob/master/languages.toml) is used.
diff --git a/helix-core/src/config.rs b/helix-core/src/config.rs
index 5bd16abd..d4ebee1f 100644
--- a/helix-core/src/config.rs
+++ b/helix-core/src/config.rs
@@ -25,7 +25,7 @@ pub fn user_lang_config() -> Result<toml::Value, toml::de::Error> {
 pub fn default_syntax_loader() -> crate::syntax::Configuration {
     default_lang_config()
         .try_into()
-        .expect("Could not serialize built-in language.toml")
+        .expect("Could not serialize built-in languages.toml")
 }
 /// Syntax configuration loader based on user configured languages.toml.
 pub fn user_syntax_loader() -> Result<crate::syntax::Configuration, toml::de::Error> {