This disables ccache in the CI when building packages. I think ccache is
not necessary since the CI will never build a package twice in the same
pipeline, and ccache dir is not preserved for use between pipelines.
There is a real time cost to compiling with ccache enabled on the
initial compile (compressing objects for cache, etc), and some testing
with building the linux-purism-librem5 kernel in the CI shows this cost
can be significant... ~160 minutes when ccache is enabled vs ~105
minutes when it is disabled with this patch.
kernel compile CI runs referenced:
https://gitlab.com/postmarketOS/pmaports/-/jobs/1087602863https://gitlab.com/postmarketOS/pmaports/-/jobs/1087956999
Some testing on my laptop using osk-sdl with and without ccache for
initial compile:
ccache disabled:
$ rm _build -r && CCACHE_DISABLE=1 meson _build >/dev/null && CCACHE_DISABLE=1 time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
7.76
$ rm _build -r && CCACHE_DISABLE=1 meson _build >/dev/null && CCACHE_DISABLE=1 time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
7.89
$ rm _build -r && CCACHE_DISABLE=1 meson _build >/dev/null && CCACHE_DISABLE=1 time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
7.71
ccache enabled:
$ rm /tmp/ccache _build -r && CCACHE_DIR=/tmp/ccache meson _build >/dev/null&& CCACHE_DIR=/tmp/ccache time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
8.62
$ rm /tmp/ccache _build -r && CCACHE_DIR=/tmp/ccache meson _build >/dev/null&& CCACHE_DIR=/tmp/ccache time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
8.46
$ rm /tmp/ccache _build -r && CCACHE_DIR=/tmp/ccache meson _build >/dev/null&& CCACHE_DIR=/tmp/ccache time meson compile -C _build
ninja: Entering directory `_build'
[10/10] Linking target osk-sdl
8.54
We may wish to distribute some files that are explicitly
non-free, and it makes sense to place them in a separate
non-free directory rather than in main.
Unmaintained devices are device packages that:
- Are known to be broken in some way without an active maintainer
who can investigate how to fix it, or
- Have not received any updates for a very long time, or
- Are discouraged from using because they are just intended for testing.
An example for this are ports using the downstream kernel for devices
which have a mainline port that is working quite well.
Unmaintained devices are still built by bpo (otherwise it would not make
sense to keep them), but they do not show up in "pmbootstrap init".
However, it is possible to manually select them by entering the name.
pmbootstrap will warn in that case.
Unmaintained packages should have a # Unmaintained: <reason> comment
in the APKBUILD, this comment is displayed in "pmbootstrap init"
so that the user knows why the device should not be used unless they
know what they are doing.
In this git repository, everything is about aports, this distinction was only
useful back in the day when pmbootstrap and pmaports were in the same git
repository.
Refactor the install code to be generic, so we only need to add the new
path in one place when adding a new file. Put the full path into the
filename to make this possible.
I've tried to mimic the final directory structure in the aport first
(e.g. $pkgdir/sbin/swapfile -> main/postmarketos-base/sbin/swapfile),
but that leads to conflicts as abuild only takes the filename for
checksums (conflict with etc/conf.d/swapfile):
https://gitlab.alpinelinux.org/alpine/abuild/-/issues/10013
We don't really need checksum verification for the files shipped in the
same directory (not downloaded), but nevertheless this is a bug. It
leads to confusing behavior and might be fixed by letting abuild demand
that there is only one source file with the same name, as I suggested in
the issue linked above. So let's avoid this altogether with the flat
file name.
postmarketos-base will soon build its source="" variable with a shell
function. Support this use case by not trying to parse source="" with
pmbootstrap anymore, but rather parsing the trivial sha512sums
variable.
Don't print a message for each APKBUILD that is getting parsed. pytest
will display the stdout below the raised exception, so with the print
one had to scroll up a lot to find the actual error message. Now it is
printed at the bottom.
When using '[ci:skip-build]' without any changed packages, do not
attempt to run "pmbootstrap checksum" without any package arguments.
This may happen, because CI in MRs runs across all patches in the MR,
but CI on branches (master, v20.05) only run on the last commit.
Prevents CI failures like this one:
https://gitlab.com/postmarketOS/pmaports/-/commit/9a48b41a2362ba0dd2b0901cae966011b37d1856
Split the aports-build job into arch-specific build-x86_64,
build-aarch64, ... jobs. Extend build_changed_aports.py to accept the
architecture as argument, and to build all packages for that arch where
possible.
Import and use pmbootstrap code for APKBUILD parsing etc, as we do this
already in various testcases running in this repository.
Co-Authored-By: Bart Ribbers <bribbers@disroot.org>
Verify _pmb_recommends for architectures listed in the APKBUILD of
postmarketos-ui-* packages, not from their binary packages. Otherwise
disabling an architecture in a merge request will not be respected, as
the binary package still has the arch enabled.
Libhybris is failing to build for armhf too. Instead of adding !armhf
everywhere, let's just remove it: as it became clear in #703, nobody is
willing to put in the effort to maintain this properly.
If you would like to continue with libhybris on postmarketOS, please
create your own aports repository and put the aports there.
Related: https://wiki.postmarketos.org/wiki/HybrisCloses: #703, #700, #678
Instead of encouraging the user to add the directory to
test_directory_structure.py, mention that we just moved the firmware to
device/{main,community,testing}/. This will be more helpful for users
who get the message after rebasing on master.
In the rare case that we need to change the directory structure, we will
figure out which file to edit.
[ci skip]: various firmware packages fail when built at the same time
(they use generic source files like "bcmdhd.cal" with different checksums)
Add common.get_upstream_branch() and use it instead of assuming "master"
as upstream branch.
Example: user forks v20.05 to v20.05_some-fix, then creates a merge
request. CI must use v20.05 as upstream branch, when comparing for
modified files etc.
This script is used by multiple repositories by now, let's make sure we
only have one place where we need to maintain it. The version in
ci-common is shellchecked and displays less output for 'pmbootstrap
init' again (unless it fails, then it shows the whole log).
Read kernel module names from files in
/etc/postmarketos-mkinitfs/modules and print out which modules are
required by which of these files as the initramfs gets generated. Put
the default modules into a new 00-default.modules. This allows mkinitfs
hooks to add modules to the initramfs.
Parse the modules from a file similar to modules-load.d, with commented
lines and empty lines ignored. Add a simple test script for that and run
it in check().
Split mkinitfs.sh into two files, so the functions (now in
mkinitfs_functions.sh) can be sourced in a test. Such a test will be
added in the next commit.
Move $BINARIES_EXTRA into the get_binaries_extra() function, so its
$(find...) does not get executed when sourcing the functions file. Move
$BINARIES into get_binaries() too for consistency.
Make it pass shellcheck by fixing obvious mistakes like '==' instead of
'='. Add "shellcheck disable" lines for things that are intentionally done
or would be a bit more complicated to rewrite. It would be nice to
refactor some things like '${outfile/initramfs-/uImage-}', but this should
be done in a separate change (patches welcome).
Remove unused variable "modules_path". While at it, add two new fatal
error messages instead of just "exit 1" without any comment and remove
redundant mkdir in APKBUILD.
At the moment the function does not handle all cases correctly.
For example, it cannot say which package was changed when a file
in a subdirectory of a package was changed.
This is used e.g. in device-samsung-golden:
device-samsung-golden
├── APKBUILD
├── deviceinfo
├── downstream
│ ├── init-usb-hook.sh
│ ├── module-config.conf
│ └── modules-load.conf
└── kwin.sh
At the moment, changing a file under "downstream" would attempt
building a "downstream" package, instead of device-samsung-golden.
Refactor the function a bit to walk up the directory hierarchy,
until we find an APKBUILD.
For APKBUILD linting we care only about APKBUILD files that were changed,
not to which package they belong or whatever.
The "with_directory" option of get_changed_packages() really meant
"just give me changed files", which means that we can just use
get_changed_files() instead.
All users of get_changed_files() check if the file was removed,
so it seems like most of them do not need files that were removed.
Also, there is no need to have files listed twice,
so we should return a set instead of a list.
It seems that the .gitlab-ci.yml did not trigger properly when the file
was changed last time, and now it does in this MR. Fix the file to make
it pass.
Do not run upstream compatibility checks whenever pushing to master.
This is confusing, because if the upstream compatibility check fails,
then it appears that the last patch was broken in some way although it
isn't related.
I've moved the upstream compatibility checks to a separate repository
already, and added a badge to the pmaports gitlab project that indicates
whether they are currently succeding or not. The checks run hourly now.
Related: https://gitlab.com/postmarketOS/monitoringFixes: #457
pmbootstrap now allows APKBUILDs in arbitrarily nested directories
within pmaports. This means that we really need some checks to ensure
APKBUILDs don't end up all over the place.
Add a test case with a whitelist of allowed directories.
Having this list in pmaports allows changing it without a new pmbootstrap
release.
At the moment we assume that all files except in /.* belong to
a package. Now that .shared-patches was moved to device/.shared-patches
that does not work correctly anymore.
This method should really check to which package the files belong
(e.g. walk up directories until it finds an APKBUILD) instead of
assuming they are directly in the same directory as the APKBUILD.
For now just ignore **/.* (i.e. all files in dot folders),
to unblock the MR since it's a critical fix.
Bart Ribbers