Managing Scala projects in Vim with Ag and Ctags
Navigate in large Scala projects easily with Silver Searcher and Ctags.
Last year I wrote a short summary about the Vim plugins I was using at that time for Scala development. Those were the low-hanging fruits I could easily configure to solve my initial comfort problems with Vim.
Since then I've attempted to tackle larger projects. There were some useful plugins in my original setup, for example:
- Tight Git integration with vim-fugitive, vim-gitgutter and vim-airline. While I mainly use the command-line to interact with Git, these plugins make diff and blame much easier and provide the necessary contextual information to tell which lines have been modified since the last commit.
- File navigation and searching with The NERD Tree and CtrlP. Exploring folder structures and finding files are as easy as it would be with an IDE.
- Scala compile error reporting with sbt-quickfix.
- NERD Commenter, neocomplete and Rainbow Parentheses Improved for enhanced editing experience. Earlier, I used the Rainbow Parentheses plugin, but in many occasions it messed up the colors of the closing brackets.
I still use the two terminal setup: one Vim session to edit the source files, and one SBT console to compile and test the code. This setup is really handy, especially because SBT can watch file changes and run the tasks accordingly. I've switched from KDE to xmonad, the tiling window manager, because its keyboard-oriented philosophy is better suited to this configuration.
But many essentials were missing. For example, when I needed to search for a piece of text in the project, I usually opened up a third terminal to use find or mc. This is very uncomfortable, and a really clumsy way of looking up method and class definitions.
Another problem is the lack of API discovery mechanisms, such as code-aware autocomplete or in-place documentation. The only help besides the compiler error messages are the ScalaDoc available online and code snippets from Stack Overflow.
In this post I briefly present some tools I've come across to mitigate these problems: Silver Searcher and Ctags.
Fast grepping with Silver Searcher - Ag
The :grep command is a convenient way to search for text occurrences in multiple files from Vim. With the Silver Searcher, the grep can be faster, while producing much saner results by respecting the .gitignore files. CtrlP can be configured to use Silver Searcher too, so it can enjoy its benefits.
As it's described in this post, it is really easy to configure, all you have to do is install the Silver Searcher:
# Ubuntu
sudo apt-get install silversearcher-ag
# OSX
brew install the_silver_searcher
and configure your .vimrc to use it:
" The Silver Searcher
if executable('ag')
" Use ag over grep
set grepprg=ag\ --nogroup\ --nocolor
" Use ag in CtrlP for listing files. Lightning fast and respects .gitignore
let g:ctrlp_user_command = 'ag %s -l --nocolor -g ""'
" ag is fast enough that CtrlP doesn't need to cache
let g:ctrlp_use_caching = 0
endif
By default in Vim, one can search for the word under the cursor with the asterisk key (*) in the current file. Going further on this line, I mapped the <leader>* to search for the selected word in the whole project.
nnoremap <Leader>* :grep! "\b<C-R><C-W>\b"<CR>:cw<CR>
This is a convenient and fast way to look up usages without requiring any language-awareness from the editor. Setting it up is a one-time activity, then you can forget about it and enjoy its benefits.
Method and class definition lookup with Exuberant Ctags
Ctags is a tool that generates a tag file by indexing the source files looking for important language constructs. The indexing always depends on the language of the language of the source file, for example it can tag method, class, and member declarations. The tag file allows Vim (and other compatible editors and utilities) to easily navigate to the tagged constructs while browsing the source code.
Exuberant Ctags is a multilanguage implementation of Ctags that was originally distributed with Vim. It supports many languages out of the box, but unfortunately Scala is not one of them. Luckily, it can be extended to support more with custom regular expressions defined in its dotfile.
3 steps are needed to use Ctags with Vim:
- install Ctags
- configure Ctags by creating the .ctags file
- configure Vim to use the tag file generated by Ctags
Then, you can trigger Ctags at any time to generate or refresh the tag file.
To install Exuberant Ctags, issue the following command:
# Ubuntu
sudo apt-get install exuberant-ctags
# OSX
brew install ctags-exuberant
Ctags can be customized with the .ctags file, located in the home folder. To enable indexing of Scala sources just add the Scala language definitons to it, as it is described in this post:
--langdef=scala
--langmap=scala:.scala
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*(private|protected)?[ \t ]*class[ \t ]+([a-zA-Z0-9_]+)/\4/c,classes/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*(private|protected)?[ \t ]*object[ \t ]+([a-zA-Z0-9_]+)/\4/c,objects/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*(private|protected)?[ \t ]*case class[ \t ]+([a-zA-Z0-9_]+)/\4/c,case classes/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*(private|protected)?[ \t ]*case object[ \t ]+([a-zA-Z0-9_]+)/\4/c,case objects/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*(private|protected)?[ \t ]*trait[ \t ]+([a-zA-Z0-9_]+)/\4/t,traits/
--regex-scala=/^[ \t ]*type[ \t ]+([a-zA-Z0-9_]+)/\1/T,types/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*def[ \t ]+([a-zA-Z0-9_]+)/\3/m,methods/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*val[ \t ]+([a-zA-Z0-9_]+)/\3/l,constants/
--regex-scala=/^[ \t ]*((abstract|final|sealed|implicit|lazy)[ \t ]*)*var[ \t ]+([a-zA-Z0-9_]+)/\3/l,variables/
--regex-scala=/^[ \t ]*package[ \t ]+([a-zA-Z0-9_.]+)/\1/p,packages/
Also, Ctags indexes everything by default, but it provides an exclude parameter to opt-out of tagging source files in certain paths. The exclude patterns can be defined in the .ctags file as well. Currently, I use the following:
--exclude=target
--exclude=.git
--exclude=.svn
--exclude=.hg
--exclude=node_modules
--exclude=bundle.js
--exclude=*.js.map
--exclude=*.min.*
--exclude=*.swp
--exclude=*.bak
--exclude=*.tar.*
Or, if you'd simply like to ignore everything listed in the .gitignore file, add --exclude=@.gitignore, but keep in mind that the indexing will fail if the file doesn't exist.
The final step to set up Ctags is to add the following to the .vimrc file to let Vim know where the tags are.
set tags=./tags;,tags;
Tag files can be generated by invoking Ctags. A nice tip from this post suggest that you should generate the tag file to the .git folder, since it is normally ignored by the VCS and most search engines, but Vim can still pick it up. Therefore, generate the index file with the following command in the root directory of the project you'd like to index:
ctags -R -f ./.git/tags .
The tag file generation is really fast, it shouldn't take too long even for larger projects. The tag file is reloaded by Vim every time a file is opened. (Or when the current buffer is reloaded, for example with :e!.) Because the tag file generation is a frequently needed task, it can be handy to create a mapping for it:
map <C-F12> :!ctags -R -f ./.git/tags .<CR>
The -f parameter ensures that the tag file can not be generated anywhere by accidentally pressing Ctrl+F12, but only in the root of a project. There are many ways to use the tags in Vim. The most obvious is to find the definition of the method under the cursor by pressing Ctrl+]. This of course works with class definitions and other language constructs too. To jump back where you left off press Ctrl+t. With :tag <tag-name> you can jump directly to a definition, or browse them with :CtrlPTag. It's useful to list for tags in the current file when looking for method and field declarations, and for this I recommend the Tagbar plugin.
An important characteristic of Ctags is that it "just" indexes the source files based on a set of regular expressions, so it's not fully aware of all the rules of the language. Ocassionally it shows a few false positive matches that you have to check, but generally it's not a big deal. For example, if you have two classes, each containing a getName method and try to navigate to the definition of one of the methods, then Vim - at best - will list both definitions that you have to choose from. Because it's really common to browse between similar tags, I've remapped my navigation to be a bit more helpful:
nnoremap <C-g> <C-]>
vnoremap <C-g> <C-]>
map <C-h> :tn<CR>
map <C-f> :tp<CR>
map <C-o> :CtrlPTag<CR>
map <C-i> :TagbarToggle<CR>
Although it requires some initial setup, navigating by tags provides really great developer experience. Having control over the tag file generation can be both a powerful feature and a footgun, so you might consider automating it for example with vim-autotag.
The setup described above is not tied to Scala, it can be used with other languages as well.
Exploring dependencies with sbt-ctags
Although the Ctags setup above is really powerful it has a shortcoming: it can't navigate to methods and classes defined in dependencies or in the standard library. The ctags-sbt plugin aims to solve exactly this problem. It downloads the source of every dependency (both Scala and Java) and indexes them with Ctags.
With a little configuration the previous setup can be augmented with sbt-ctags, as it is described in its Github page. Just create the ~/.sbt/0.13/plugins/plugins.sbt file with the following contents:
resolvers ++= Seq(
"Sonatype OSS Releases" at "https://oss.sonatype.org/content/repositories/releases/",
"Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/"
)
addSbtPlugin("net.ceedubs" %% "sbt-ctags" % "0.2.0")
Then the tag file can be generated with sbt gen-ctags. Beware though, depending the size of your dependencies this tool can create enormous tag files. I've tested it on Time-admin, a small time-tracking web application consisting of about 50 classes. While the tags of the Time-admin sources are below 2 MB, the tags of its dependencies are more than 100 MB. The file generation can be slow (especially the first time as the sources have to be downloaded), but the main problem with this size file is that it kills CtrlPTag on older machines. In this case, instead of fuzzy tag finding use :tag <tag-name>. With the tab key the tag name can be auto-completed, so while it's a bit less powerful, it provides some browsing capabilities.
To ease these problems, I've configured sbt-ctags to use a separate file for the index, so it doesn't write the previously generated tag file over. To achieve this, create a file to ~/.sbt/0.13/sbt-ctags.sbt with the following contents:
import net.ceedubs.sbtctags.CtagsKeys
CtagsKeys.ctagsParams ~= (default => default.copy(tagFileName = "./.git/tags-dep"))
And modify your .vimrc to use this additional tagfile:
set tags=./.git/tags-dep,tags-dep,./.git/tags,tags
This is convenient as the tags for the project sources can be generated independently from the dependencies that change only once in a while.
Summary
Silver Searcher and Ctags are powerful tools that can be integrated nicely with Vim, providing simple and fast means to navigate effectively in larger projects. I'm using them for a while, and the difference they made is really considerable. Despite their simplicity and the fact, that none of them has complete understanding of the semantics of the programming languages, they work surprisingly well.
They can be used with many programming languages, so I recommend giving them a try even if you are not programming in Scala. With a minimal setup they can be great enhancements to anyone's Scala toolbox who likes to program in Vim.
