NOTES ON THE EMACS PACKAGE ARCHIVE The GNU Emacs package archive, at elpa.gnu.org, is managed using a Git repository named "elpa", hosted on Savannah. To check it out: git clone git://git.sv.gnu.org/emacs/elpa cd elpa make setup That leaves the elpa/packages directory empty; you must check out the ones you want. If you wish to check out all the packages into the packages directory, you can run the command: make worktrees You can check out a specific package into the packages directory with: make packages/ Changes to this repository propagate to elpa.gnu.org via a "deployment" script run daily. This script generates the content visible at https://elpa.gnu.org/packages. A new package is released as soon as the "version number" of that package is changed. So you can use 'elpa' to work on a package without fear of releasing those changes prematurely. And once the code is ready, just bump the version number to make a new release of the package. It is easy to use the elpa branch to deploy a "local" copy of the package archive. For details, see the README file in the elpa branch. ELPA PACKAGES BUNDLED IN THE DISTRIBUTION TARBALL Why Bundle? - To provide backward compatibility for users while allowing more flexible releases, and more convenient maintenance. - ada-mode was originally included in the emacs tarball, but was removed when the ELPA version was capable enough. Some users complained; bundling ada-mode would (almost) restore the original behavior. - org and other packages currently maintain two copies of their code, in emacs.git master and elpa.git. Synchronizing them is difficult; it is not a simple `git push/pull'. Bundling would eliminate the copy in emacs.git. Users do not need to do anything special update a bundled ELPA package; the normal package.el mechanisms recognize all the versions properly. Emacs core code may not depend on (bundled or non-bundled) GNU ELPA packages unless they are ":core" packages (ie, they have code in both emacs.git master and elpa.git); if that functionality is required, the package must be moved to Emacs. Rationale: - A simple git clone/checkout of emacs.git master creates a workspace will all code required to build and run Emacs. - Any ELPA package can be changed without affecting Emacs core. For single file packages, the file can simply be in both elpa.git and emacs.git master. The package is marked ":core" in the ELPA "elpa-packages" file. For example, eldoc and flymake are :core packages. This mechanism is not supported for multi-file packages, because maintaining sync between the files in emacs.git and elpa.git is more problematic - with the mechanism described below, you can use git push/pull to maintain sync. Every GNU ELPA package has a corresponding branch `externals/[PKGNAME]` in elpa.git. To bundle a package in Emacs, add to elpa.git a branch `emacs-NN/[PKGNAME]` which holds the version to be included in the next Emacs release of version NN. `make dist' would then grab the bundled packages's code from those branches, and include them in /usr/share/emacs/NN.MM/elpa/* in the distribution. At Emacs startup, the bundled `elpa' directory is added to package-directory-list, and is treated the same as other entries in that list. For Emacs developer convenience, there is a script `checkout-bundled-elpa' that checks out the correct versions of the bundled ELPA packages into an elpa worktree. In addition, there is an emacs function `add-bundled-elpa-packages' that adds those package directories to `load-path', so they work as if they were simply in the emacs.git worktree. Both of these features are optional. UNRESOLVED ISSUES: - There are currently (jan 2020) some multi-file packages with code in both elpa.git and emacs.git; org, gnus for example. Once the ELPA bundling process is worked out, it is expected that those packages will use the bundling process instead. - In the future, for packages like org that used to have code in emacs.git master and elpa.git, the emacs.git checkout no longer has the package, which is not backward compatible for developers. Emacs developers who want those packages now have to install them via `list-packages', or check them out from elpa.git. - Should fetching a worktree from emacs.git also fetch the bundled GNU ELPA packages (into the Emacs worktree or somewhere else)? - Not fetching ELPA packages into the emacs.git worktree helps enforce the rule that Emacs core code may not depend on ELPA code. - Not fetching ELPA packages means all packages are treated equally; Emacs developers do not have to be aware of whether some packages are bundled or not. Only the package developer has to know. - Fetching them makes it clearer what's bundled and what isn't. - With the baseline mechanism, developers who want a bundled ELPA package checkout that package from elpa.git and either install the package from there or include elpa/packages/[PKGNAME] in their `load-path', as for any (non-bundled) ELPA package. - An alternative would be to declare a git submodule for each bundled package, with a corresponding branch `elpa/[PKGNAME]` (instead of the branches 'emacs-nn/[PKGNAME]'). The submodule declaration would place the packages in emacs/elpa (parallel to emacs/lisp), so Emacs subdirs.el does not see them. emacs/elpa is in package-directory-list by default when running from the source directory. - If we have submodules, new developers or a fresh checkout must use: git clone .../emacs.git --recurse-submodules for the initial clone. Ideally, the Savannah web page https://savannah.gnu.org/git/?group=emacs would say that. - Other variations on where to place the submodules in the emacs worktree offer more backward compatibility. - One issue is whether the autoloads of bundled ELPA packages are processed when we dump Emacs (like we do for all the packages that are in Emacs core), or whether that's done during `package-activate-all` (i.e. between `early-init.el` and `init.el`). Doing it at dump time gives better startup times, at the cost of making it impossible for the end-user to prevent activation of a package (they can still undo the activation after the fact, of course, but that needs to be done in ad-hoc ways). Users who care about startup time can do their own dump. - Using submodules would help automate cleaning and updating the bundled packages; with the baseline mechanism, developers must separately manage the elpa worktree. - Automating compiling C, Ada, Rust or other language code required by the package, to executables or modules. This includes defining standard places to put executables, modules, and other aritifacts. This is more important for bundled packages; users expect Emacs to "just work" out of the box. -- updated 20 Jan 2021