Dev setup

Mac & Linux

1. On Mac:

   brew install git just sqlx-cli mdbook postgresql shellcheck tokei ripgrep bat exa procs fd sd cue-lang/tap/cue
   npm install -g zx
   

   On Linux:

   cargo install sqlx-cli mdbook just tokei ripgrep bat exa procs fd sd
   sudo apt-get install git shellcheck postgresql
   brew install cue-lang/tap/cue
   npm install -g zx
   

2. Install mkcert: https://github.com/FiloSottile/mkcert

   1. Make sure to run mkcert -install to install the dev CA, but no need manually generate any certs.

3. Install GitHub Desktop: https://desktop.github.com/

4. Install Node

   1. Install nvm: https://github.com/nvm-sh/nvm

   2. Install specific versions of Node and NPM:

      nvm install v16.14.2
      nvm alias default v16.14.2
      npm install -g npm@8.5.5
      node --version && npm --version
      

5. Install Rust

   1. Install rustup: https://www.rust-lang.org/tools/install

   2. Install components:

      rustup component add rustc cargo rust-std rustfmt clippy rust-docs
      

   3. Check that Rust is >=1.62.0:

      rustc --version
      

6. Install the Protocol Buffers compiler:

   1. Manually download the binary (brew and apt for protoc are pretty out of date):

      cd ~
      # This URL is for Intel Macs - for other machines check https://github.com/protocolbuffers/protobuf/releases
      curl -o ~/protoc.tar.gz -L https://github.com/protocolbuffers/protobuf/releases/download/v21.1/protoc-21.1-osx-x86_64.zip
      mkdir -p ~/protoc
      unzip protoc.tar.gz -d ~/protoc
      rm ~/protoc.tar.gz
      

   2. Add the following to your .bash_profile / .zshenv / etc:

      export PATH="$HOME/protoc/bin:$PATH"
      
      # meant to be consumed by the `prost-build` Rust crate:
      export PROTOC="$HOME/protoc/bin/protoc"
      export PROTOC_INCLUDE="$HOME/protoc/include"
      

   3. Check that protoc is >=3.21.1:

      protoc --version
      

7. Install Docker Desktop for Mac: https://docs.docker.com/docker-for-mac/install/

   1. Check that Docker is >=20.10.11 and Docker Compose is >=1.29.2:

      docker --version && docker-compose --version
      

   2. This step is not needed for Linux, as resources are already unlimited.

      Increase the system resources granted to Docker. This is necessary because compiling release binaries in Rust is very resource-intensive — especially compiling & optimizing a large number of dependencies. Example configuration:

      

Clone + Build + Run

1. Create a GitHub personal access token per these instructions.

   • For scopes, all you need is the repo group.
   • Keep the personal access token in something like 1Password.

2. Clone:

   mkdir -p ~/git && cd ~/git
   git clone https://github.com/realismlabs/dataland
   git clone https://github.com/realismlabs/dataland-infra
   git clone https://github.com/realismlabs/dataland-engineering-handbook
   

   • When asked for a username, use your GitHub username.
   • When asked for a password, use your personal access token, not your GitHub account password.

3. cd into dataland repo:

   cd ~/git/dataland
   

4. Build and run:

   just setup
   

5. Go to https://app.dataland.land - if everything works then the setup is complete.

VSCode

1. Install VSCode: https://code.visualstudio.com/

2. Go to Extensions and search for @recommended. Install all the Workspace Recommendations.

   • See here for details.

3. Bring up the Command Palette (Shift+Command+P) and run:

   Shell Command: Install 'code' command in PATH
   

4. Open the Dataland repo in VSCode:

   code ~/git/dataland
   

5. Check that the Rust extension is working. Bring up Quick Open (Command+P) and open:

   dataland-server/main.rs
   

   Clicking through symbols (Command+Click) to their definition sites should work.