Codesigning everything and combatting malicious code

[Jason Liu]

Apologies if what I'm about to say seems tangential... hopefully it
provides historical context that is useful to the discussion at hand.

=====

Remember that code signing was first implemented for apps that were
published on the iOS App Store. This was very quickly (maybe even
simultaneously?) adopted for the Mac App Store. Since the code signing was
generally seen as "a good thing"™ in terms of security, Apple has since
rolled it out nearly everywhere. Some people, including me, have seen this
as just one more indication of the "iOS-ification" of the Mac operating
system, a process which began when they renamed Mac OS X to macOS. Another
example of the "iOS-ification" of macOS is where, starting in Catalina, and
then in Big Sur, Apple moved the System volume to a partition that is
mounted read-only, and then also sealed that volume into a Signed System
Volume (SSV). (See? there's that word again... "Signed".) This structure of
a read-only and signed volume that contains the entire operating system is
the way that iOS has been structured for a very long time (maybe even since
the beginning). Getting around this structure, and turning the volume where
iOS resides into one that is read-write, is one part of what is generally
referred to as "jailbreaking" iOS.

Apple's app stores are highly centralized, being 100% hosted, administered,
and controlled by a single entity. Thus, they have a need for code signing,
so that they can use the code-signing keys as a mechanism to control
whether or not a third-party developer's app is allowed to be visible to
the public through their (first party) app stores. On the other hand,
MacPorts is much more similarly structured *—* in terms of infrastructure,
governance, and philosophy *—* to a package manager on Linux, and is highly
decentralized. This sentiment appears to be echoed in Wikipedia's article
on code signing (https://en.wikipedia.org/wiki/Code_signing), in the last
paragraph of the section entitled "Providing security". In particular, I
would like to point out the citation at the end of that paragraph, which
links to a discussion of apt-secure on the Debian wiki (
https://wiki.debian.org/SecureApt). If you read through the Debian wiki
page, you will see that, circa 2003-2005, when Debian moved away from
simply using MD5 checksums, they decided to go the route of using public
key cryptography in the form of GPG keys, instead of going down the road of
code signing the way that Apple has implemented it.

=====

On Tue, Mar 22, 2022 at 12:02 AM Joshua Root <jmr at macports.org> wrote:

> On 2022-3-22 12:20 , Ryan Schmidt wrote:
>
>> Could MacPorts codesign everything installed by ports? If so, should we?
>> What benefits would that bring? How would we do it?
>
>
> We could ad-hoc codesign everything, which would not improve security at all,
> but would get GateKeeper to ease up a bit on restrictions on incoming
> network connections and the like.


If ad-hoc code-signing everything would not involve the need for anyone,
either on the MacPorts side or the user side, to have a Developer ID, then
it seems like this would generally be a good idea. Reducing the number of
interactions with GateKeeper would help reduce user confusion. (Does anyone
remember the frequency of the UAC prompts in Windows Vista?)

Doing actual useful codesigning would require a few things:
> 1. A Developer ID for the project. I'm happy to sign the installers with my
> personal Developer ID because I build them myself and I am familiar with
> the code contained in them. I would not sign arbitrary third party code.
> 2. A Developer ID for every user who needs to install any ports that are not
> available as binaries. We obviously can't distribute our secret key to
> the public, so anything built locally needs to be signed locally, with a
> locally configured identity.
>

This seems like it would be adding a huge burden onto people who simply
want to use MacPorts.


> 3. A willingness to endorse every binary we ship by putting our signature
> on it.
>

Does MacPorts, as a group of developers, really want to be in the business
of endorsing every binary that's available in our catalog of nearly 30,000
packages? The various Linux package management systems certainly don't make
any such claims or guarantees, and in fact explicitly state the opposite.
And even though Apple also has text stating that they don't "endorse" any
apps in their app stores, the fact of the matter is that they do still take
it upon themselves to police the apps (and the developers) on their app
stores. This also relates back to point #1, where the MacPorts Developer ID
would be getting used to arbitrarily sign third-party code.

4. A plan for what to do if we inadvertently ship malware and our Developer
> ID certificate is revoked. AIUI, that would make it impossible to run
> anything signed with the existing certificate if GateKeeper is enabled.
> Everything would presumably have to be re-signed and reinstalled.
>

I believe Josh is correct. A certificate revocation means having to start
over from scratch and "re-publishing" everything with a new certificate.
This is also what happens when a developer's certificate gets revoked on
any of Apple's app stores.

=====

Finally, I'm of the belief that code signing would not prevent, nor was it
designed to prevent, malicious or buggy code from making its way into
distributed software. It didn't prevent the Log4j bug/vulnerability, and it
wouldn't have prevented the node-ipc situation from occurring. In my
opinion, it's not, nor should it be, our job to guarantee that we are
providing bug-free and non-malicious software from upstream third-party
sources. Our current merge review process already implements many of the
best practices used by open-source projects to try to guard against such
software from making it from upstream all the way to being released by our
distribution network. However, I believe that it IS our role to strive to
provide bug-free and secure first-party software, i.e. MacPorts base,
MacPorts legacy support, Portfiles, and the other pieces of code that are
directly released by the group of volunteers that make up "The MacPorts
Project"™.

I'm only a package maintainer, and so I don't expect my opinion to have
much weight on final decisions, but I'm of the opinion that the current
situation should stay as it is: we release a MacPorts installer that has
been code signed, but we should not be code signing any of the ports using
a Developer ID certificate. Ad hoc signing of ports might be useful, as
long as it doesn't involve the need for a Developer ID certificate.

-- 
Jason Liu


On Mon, Mar 21, 2022 at 9:21 PM Ryan Schmidt <ryandesign at macports.org>
wrote:

> Sorry, this got long.
>
> I want to move a discussion to the macports-dev list that began with a
> user's question to macports-mgr. This user ran a third-party utility which
> reported that some files installed by MacPorts were not codesigned and
> asked if it was a concern.
>
> I replied that the files installed by the MacPorts installer are
> codesigned but that MacPorts predates the existence of codesigning and
> nobody has yet added code to MacPorts that arranges for files installed by
> ports to be codesigned. Individual ports might codesign their files if
> that's how their build system was written. On Apple Silicon all (code?)
> files must be codesigned and the linker (?) takes care of ad-hoc
> codesigning the files automatically. I also said I wasn't really clear on
> what purpose codesigning serves; MacPorts and macOS got by fine without it
> for years.
>
> The user replied asking if codesigning might reduce the risk of
> compromised open source packages, referencing this incident in which the
> developer of node-ipc deliberately released a version containing malicious
> code:
>
>
> https://www.schneier.com/blog/archives/2022/03/developer-sabotages-open-source-software-package.html
>
> This is a hypothetical question since node-ipc is not in MacPorts and as
> far as I know a similar situation has not happened with any software that
> we do have in MacPorts. And since then node-ipc appears to have been forked
> by a different developer and a new non-malicious version has been released.
>
> I felt that the broader MacPorts developer community might care to hear a
> reply to this or to discuss whether and how we should modify MacPorts base
> to codesign everything installed by ports.
>
> ~
>
> Certainly the situation with node-ipc was undesirable but I don't see how
> MacPorts codesigning its files would solve it. I'll talk about that in a
> minute. First let me discuss what safeguards MacPorts already contains.
>
> MacPorts ports already fetch specific maintainer-tested versions of the
> source code. If a developer releases a new version, nothing changes in
> MacPorts until a MacPorts contributor updates the Portfile to use that new
> version. Before doing so the MacPorts contributor is expected to verify
> that the port at least compiles on one macOS version; that the port's test
> phase runs, if the port has one; that "port lint" doesn't show any easily
> fixable mistakes; and ideally that some basic functionality of the
> installed files works. If the port has subports, this applies to all
> subports as well.
>
> MacPorts already checksums distfiles. The MacPorts master build server
> saves a copy of distfiles (assuming the checksum matches at the time) and
> these are mirrored on servers around the world. If a developer replaces a
> distfile on their server with something else (maliciously or not), MacPorts
> will not install the port using that distfile; it will issue a checksum
> mismatch error and tell the user how to report it to us. Or MacPorts might
> download the previous "good" file from a MacPorts mirror, bypassing the
> developer's new replacement file. When we notice such a "stealth update"
> has occurred, proper procedure is to compare the old and new file to see
> whether something malicious has happened so that we can decide whether we
> should modify the port to use the new file or to bypass it. (We might also
> bypass a new developer distfile if it contains no relevant differences from
> the old one.)
>
> Ports that fetch their sources from a revision control system do not enjoy
> the protection of checksums. Although ports that fetch source from a
> revision control system specify which tag or commit hash to fetch, it is
> conceivable that a developer with sufficient access to that repository
> could delete an old tag and replace it with a new tag of the same name that
> contains different software. This is one of the reasons why we recommend
> ports fetch using distfiles, and the vast majority do. We might consider
> recommending that ports that fetch directly from a git repository
> (fetch.type git) never use a tag, and always use the commit hash
> corresponding to that tag, since replacing the contents of the repository
> while keeping the same sha1 hash is, as far as I know, still impossible in
> the general case. (Yes, it is possible to engineer a sha1 collision, but
> only if you can carefully control both the old and new files. Generating a
> sha1 collision against some existing old file is a very different matter.)
>
> Some ports' build systems fetch additional files from the Internet at
> build time. This too is discouraged in MacPorts. Who knows whether such
> build systems download specific versions of those files or whether their
> integrity is verified in any way. This represents another avenue by which
> unverified files (which might have been replaced with malicious versions)
> could make their way into a build of a port. And if that one server that
> the build system is programmed to download from is down, the installation
> fails. Ports should specify all their needed files as distfiles and allow
> MacPorts to be in charge of downloading and mirroring and checksumming
> them. To my annoyance, developers are increasingly adopting build systems
> that download their own files. The difficulties that this poses for
> verifying the integrity of the final product should be communicated to
> those developers by interested MacPorts contributors. Developers should
> continue to employ the time-tested practice of offering a single distfile
> that contains all the code needed to build the project (excluding external
> libraries).
>
> For most ports, most users can receive binary archives from us and do not
> need to obtain the source code and compile it themselves. These archives
> are produced on build servers that I control; nobody else has access or
> could compromise them (except for the Apple Silicon build machine which is
> colocated in a reputable data center). All build machines transfer the
> archives they build to my master build server which signs them with our
> private key. Nobody has access to this private key except me at the moment.
> Prior to 2017 when our build servers were hosted by Apple, Apple server
> administrators had access to the private key. Just like with the distfiles,
> the archives and their signatures are mirrored on servers around the world.
> When MacPorts on a user's computer downloads a binary archive it verifies
> the signature using our public key before installing it.
>
> So nobody could replace distfiles or binary archives on any server without
> error messages appearing that would stop an installation.
>
> The ports tree that users get from our rsync server as a tarball is also
> signed with our private key and verified with our public key, so a
> malicious mirror operator could not, for example, replace Portfile code
> without verification of the tarball failing.
>
> ~
>
> That brings us to node-ipc. On March 7 a developer committed this
> malicious change to node-ipc:
>
>
> https://github.com/node-ipc/node-ipc/commit/847047cf7f81ab08352038b2204f0e7633449580#diff-c2dd3b497ae886cfb8f5bf8c66c649fc2ae4afaa6660d9bbf3105d69884679c6
>
> and version 10.1.1 was released shortly thereafter. We don't have node-ipc
> in MacPorts but if we did and if someone noticed that this new version was
> available, they would have updated the Portfile's version line, fetched the
> new distfile, updated the Portfile's checksums to match it, and tried to
> install the port. They might have run the test suite and/or verified the
> basic functionality of the files that got installed. Then they would have
> committed the update. They would most likely not have known about or
> noticed the malicious code since it was programmed to take effect only
> under certain conditions.
>
> So now the checksums of the malicious distfile would be in MacPorts.
> MacPorts servers would mirror the malicious distfile. MacPorts build
> servers would build it and create binary archives of the malicious version
> which would be signed with our public key. If MacPorts contained any code
> that codesigns files, it would codesign the malicious files before they
> went into the binary archives. So I don't see how the addition of
> codesigning to MacPorts would prevent malicious code from getting into
> MacPorts via this method.
>
> It is not MacPorts policy to require contributors to examine every
> upstream change between the old software version and the new one in order
> to verify that it is free from malware or for any other purpose. I don't
> think anybody could be expected to do that. I would guess most MacPorts
> contributors are not familiar with the upstream code of the projects they
> maintain, may not even be proficient in the programming language they're
> written in, and might not be able to recognize a malicious change if one
> were there. Some projects do not even have public source code repositories
> so you couldn't review each individual commit and would have to examine a
> single aggregate diff between an old and new version which could be tens or
> hundreds of thousands of lines long; asking a MacPorts contributor to
> ascertain whether such a diff contains malicious changes seems like an
> impossibility. (node-ipc does have a public repository, and it is a small
> project and the diff between versions 10.1.0 and 10.1.1 was under 500
> lines, so a review of that change would have been possible.)
>
> It is not even our policy that contributors must read the NEWS or
> CHANGELOG files to see what changed, though I sometimes do this as it helps
> me discover things like if dependencies need to be changed. This malicious
> change in node-ipc was obfuscated so the code's purpose could not be seen
> at a glance. The change was in a JavaScript file and JavaScript files are
> often "minified" (obfuscated in order to reduce their size for lower
> bandwidth use) so this might not have been seen as suspicious. The
> description of the change in the commit message did not seem suspicious,
> and that project does not maintain NEWS or CHANGELOG files, nor would I
> imagine that a malicious change would be described as such in them if it
> did. Seems like more and more projects are ditching NEWS and CHANGELOG
> files. Some developers suggest reading the commit messages between versions
> as a substitute, which strikes me as inadequate.
>
> ~
>
> So what can we do to combat malicious code in upstream projects? Remove it
> as soon as we learn about it. Revert the port back to the previous good
> version. Or patch out the malicious code. Write a comment in the port
> advising future contributors to exercise special care when updating it and
> to actually check the differences for potential malware. If the software
> developer is an ongoing threat consider removing the port from MacPorts or
> switching to a fork by a different developer if that exists.
>
> I'll also say that all contributions to MacPorts from the public are
> reviewed by trusted contributors. Proposed changes are submitted either as
> an attachment in a Trac ticket or as a pull request on GitHub, where
> potential problems can be identified and fixed prior to those changes being
> merged into the repository and becoming available to everyone else. Pull
> requests automatically run continuous integration. Currently we check that
> the port builds on two macOS versions and run it through "port lint".
> Additional checks could be added. For example, if there is a tool that
> checks a directory of files for known malware, we could run that.
>
> Once the changes land in the repository, an email of the changes is sent
> to the macports-changes mailing list where I and others review them again.
> All MacPorts contributors are encouraged to subscribe to the
> macports-changes mailing list to see what's changing in MacPorts, to learn
> common Portfile development techniques, and to help spot and fix mistakes
> before they are noticed by our users. This review process would not have
> caught an update of a hypothetical node-ipc port to 10.1.1 since the bad
> code was not in MacPorts but in the upstream project, but it constantly
> helps uncover and fix other issues.
>
> ~
>
> Could MacPorts codesign everything installed by ports? If so, should we?
> What benefits would that bring? How would we do it?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20220322/5d058682/attachment-0001.htm>