Thoughts on Python's exceptions, and why I think they're a nightmare.
Published on
I use Python a lot at $WORK
(despite me saying I don't like Python, but it's a necessary evil), and I still feel like Python has one glaring issue that is never addressed: the actual problems with exceptions.
Exceptions are a computer science 102 or 102 topic about a special programming ability to handle errors gracefully and execute code based on given error classifications. The trade-off is that when you write a function that can potentially yield errors, the compiler must know ahead of time what classification of errors to expect, so reporting and handling goes smoother.
For instance, it would make sense if using file I/O code gives you back an error related to I/O, and contains information about what went wrong. This way, when you catch an exception, you can terminate the program and print out what happened, like the file doesn't exist, permission error, or maybe a disk error occurred that the program can't solve.
An example of this in Java would be a function header like the following:
public boolean writeToLogFile(String[] data) throws IOException {
// ...
}
The function header indicates that it can yield an error of the IOException
class, so now in your writeToLogFile()
code, should you encounter an issue writing to file, the IOException
class can be wrapped around at the call-site for the developer to implement more easily. The compiler is happy with this because should some other exception arise, it will produce a warning indicating that there's a conflict in the exception-raising type.
The bad part about this is that programmers get lazy, and this leads to lazier, poorly-written code with a lot of unnecessaries.
When an exception is thrown, it has to be caught, otherwise a runtime error will occur and produce a stacktrace dump, which isn't pretty and doesn't help the end user debug what went wrong. So we need a syntax to be able to receive exceptions when they occur, bind it to a name, and direct the error to the correct place (like a file dump somewhere), or output the error in a human-friendly way for the users or developers to interpret and benefit from.
try {
writeToLogFile("Write me!");
} catch (IOException e) {
System.out.println("I/O Error!");
}
That's great and all, but where's the lazy part come in? What about this makes coders more lazier? The fact that there is no real theoretical limit to how much you can stuff within a single try
block of code. You can put everything into a try
block.
try {
conn = openNetworkPort(8088);
incoming = conn.listen();
writeToLogFile(incoming.toString());
} catch (NetworkException neterr) {
// ...
} catch (IOException ioerr) {
// ...
} catch (Exception exc) {
// ...
}
You can stack catch
blocks to cover even more exception types, however, notice how the Exception
class itself may or may not catch any exception not covered? That's because by design, the Exception
class is the top-level class of all exceptions, aside from Throwable
which describes a data-type usable with the throw Error
syntax of Java. Any exception in the game is an Exception
class, which is one of the weakest points of the exception system.
I've talked a lot about Java, but that's really where it all starts from. The C language does not have exceptions, and most if not all C developers would prefer error codes with useful error structures. C++ has exceptions, but again, most if not all C++ developers would tell you not to use them, as they lead to performance degradataion. Java is really where they took off in popularity, and they also exist in other languages... Like Python.
The reason I don't like the try/catch
model is that it leads to a everything and the kitchen sink mentality. You can throw as much code as humanly possible in a try
block and catch only one exception. You might consider adding more exceptions later, but more likely than not, you won't, and neither will other developers. Error handling isn't sexy.
The Java way of declaring a special class of exceptions is a nice way of writing effective, developer-friendly code, but in Python, it's not exactly the same way. As a dynamic language, Python doesn't have a compiler and instead only cares about the runtime. Therefore, there is no way to get common compile-time gotchas as you would with static languages.
def write_to_log_file(data):
with open("log.txt") as f:
res = f.write(data)
if not res:
raise IOError("Write fail")
return
The Exception class type isn't lifted to the header, and instead you can yield any error you like. This raises some potential headaches, because you could be hundreds of lines of code deep in a function before you decide to throw an error. It could be less obvious than my silly little example above. But the only way you'd know if an exception ever gets raised in Python code you can't visibly see or have spotty/crappy documentation of, would be to actually just run it, observe an error, and adjust the code. No need for long compile times, just go go go.
From a developer perspective, if I had to bring into my codebase write_to_log_file
, I wouldn't even be necessarily aware it raises any kind of exceptions, at least until I ran it once in testing trials. The lack of compiler information disregards hinting like that, so there's really no way I would know. That means you can write perfectly valid code, but it's not valid in real use.
write_to_log_file("Hello!")
The above code alone, without any surrounding try/except
, is legal syntax. You just don't handle the exception. So my question to ask: is this actually good?
If you're an experienced Python developer, you're probably used to shoving lots and lots of code inside a try/except
, same as a Java developer might for very boring, uninteresting code that yields a lot of errors. The problem is, Python documentation may or may not put what exceptions are generated in the documentation. Java will. By design, Java has to include what the exceptions can be from a given function, since it is a compiler-based language and not a dynamic runtime one like Python.
This means that, should a function be used and the error goes unhandled, the compiler throws an error, and the developer must fix it's usage to some degree. Python does not do this, and makes it very difficult and time-consuming to resolve. You write code, you run it, you check for errors in the stacktrace, then fix it, run it again, and observe for errors. Some may not find themselves stuck in such a poor cycle of code-test-fix-repeat, but it is one that can happen a lot when developing big Python codebases.
So let me look at one issue with a Python function header, and one way you can maybe resolve it.
Here is a function, called write_to_log_file()
.
def write_to_log_file(data):
with open("log.txt") as f:
res = f.write(data)
if not res:
raise IOError("Write fail")
return
You may have noted above how this produces an IOError
should there be any kind of problem with writing to the file. This function can produce another exception outside of the scope of the one we wrote here: actually opening the file. Should the file fail to be opened, an exception gets thrown. We don't even handle that ourselves with our own code here, that's done by the with/as
syntax. Should we wanted to wrap that problem, we would need a try/except
around our with/as
!
That aside, nothing about the function header really shouts an exception is yielded. If you brought this into your codebase, you naturally assume it works, without errors, which is a fallacy in itself: you should probably not really trust code you don't know about to not yield errors. Any code can yield an error; it's just a little weirder in Python. If it were up to me, I'd implement a try/except
somehow into the language naturally, as to avoid developer attrition.
One way of making Python function headers more like Java would be to use the typing
library, which tries to make amends to Python developers everywhere who loathe that Python is a dynamic language. Python typing
is akin to adding type hints to the language to help indicate what the expected output should be.
def do_thing(x: int, y: str) -> bool:
return True
The added types here can reflect base Python types like str
, int
and bool
, or they can be specialized unions of a collection of types instead, like a union of all number values to create a Number
typeclass.
Number = Union[int, float]
def do_math(a: Number, b: Number) -> Number:
return a + b
But, this does not cover exceptions! At all!
This would have been the perfect case to cover exception checking to be able to type hint towards exceptions, but apparently, errors are not covered by the typing
library. Due to the dynamic-ness of Python, it does not seem like there's a sure-fired way of checking Python code for all cases of exceptions it can throw, other than extreme fuzzy-testing.
Type hinting is nice, and I would deeply encourage the use of mypy
, but since it's not baked into the language by default, it may get ignored by many.
I come from a varied background in other languages, and I can say that I feel like Python is one of the most tiring languages out there. The syntax itself, while it's fine on the surface, ends up being needlessly annoying at it's lowest points. The exception handling part is annoying, because there's few ways to minimize the impact of them without having to rewrite a lot of functions and control flow.
It's possible to be a language that doesn't rely on exceptions, but then you rely on compiler magic to pick up a lot of the pieces, which Python cannot hope to do. Some languages, even if they're compiler based, don't even necessarily rely on exceptions, but work around incorporating it as a natural data type instead. Rust, for example, does not have exceptions that interrupt the entire program; they rely on abstract data types that incorporate information into a type that you can use plainly.
Here is some code that demonstrates the use of a Result<T, E>
data type, where T
is the value we want, and E
is an error value indicating a bad operation.
fn divide(x: i32, y: i32) -> Result<i32, std::string::String> {
if y == 0 {
return Err("Division by zero.".into());
}
return x / y;
}
fn do_code() {
let x: i32 = 100;
let y: i32 = 50;
let z: i32 = divide(x, y).or(0);
}
That's it! The division by zero is an illegal mathematical operation leading to a NaN
type value, and in most languages will produce an exception. However, we produce a Result<i32, String>
type from the division()
function instead. Why? Because:
.or()
, which gets us the good result, or it provides us with whatever we fed into the .or()
function, in this case it was a plain zero.We can choose to panic, but we didn't have to; instead we safely ignored a division-by-zero error and let it be zero. Even if it threw an error. We have the choice to do something about it, or do nothing at all. There was no try ... catch
syntax here, no extra indentation, no confusing blocks or exception unpacking. This code should (hopefully) never crash on any platform it runs on. No program panics or runtime errors.
This, by all means, is pretty cool. You can encapsulate the error in a plain data type, and your code branching is no different than normal code. There's no try ... catch
special syntax that has to test for all possible error conditions; it's more functional and practical. Rust got this right in my opinion (but it was derived from Haskell).
The way I see it, the best way to work with exceptions in general, would be to:
Exception
base typesThe third line is where I will draw some controversy; why shouldn't third party APIs present exceptions to the developers/users? There are some natural problems that will occur that the user needs to be aware of, but I don't necessarily think that exceptions are the way. Your documentation won't acknowledge it by default, there is no automatic way of Python checking exceptions for you, and the code-try-fix-repeat cycle is tedious. If I had a choice, I wouldn't design an API that intentionally presented users with exceptions. There are better ways of going about things.
Here I'll present two patterns I see mostly all the time. Some libraries and packages already use these patterns, and there's nothing absurdly crazy about them. You've most likely seen them already, but never gave them much of a second thought.
The most common, boring pattern, is what I'll call "If Not None". What I mean is a common Python pattern like this:
result = do_computation("/api/get/by-id/12345")
if result is not None:
# do computations
some_func(result)
# ...
The do_computation()
has an important job in that it should, at some point, return a value, or return None
. Returning None
is implicit when you pass a return
keyword at the end of a function with no implied value, so any function that either has an empty return
, or simply doesn't have a return
at all, implies the return
value is None
. Examples are:
# returns None
def do_nothing():
pass
# also returns None
def do_nothing_again():
return
# again
def do_nothing_lastly():
return None
It's pretty natural to type return
as a way of bailing out of a function, and it's even lazier to type return
alone with no None
typed out, so this should be a pretty common pattern you've most likely seen. In fact, I'd reason that almost all functions should consider returning None
as a means of implying the existence of failure somewhere, but that probably doesn't stand to be useful when you know Boolean values are also a helpful thing.
So, what does this pattern do for us? Well, it helps us to avoid having to deal with exception mangling. Should you write some code in a library, if you handle the exceptions under the surface and present data, or return None
, the return value being None
may imply something went wrong. None
comparisons are fast, since None
is a value held in constant space by the Python runtime, so it amounts to a simple pointer comparison. None
isn't a very useful value for presenting errors, but it's a simple pattern to cover many cases of where things go wrong. For data querying, if you expect a JSON blob over the wire but get None
instead, then something happened elsewhere. You won't get an exception there, and are better off looking for what happened in your network calls.
The idea of returning None
traditionally comes from a functional language design. Similar to how Haskell has Maybe a = Just a | Nothing
, or Rust has Option<T> = enum { Some(T), None }
, or even how Lisp has 'nil
, the idea of nothing alone is in some cases a good idea, and I think Python thought hard and long about how to make Python appeal to many Lisp programmers to get them away - that's it, make an easy way of saying nothing. If nothing is returned, then nothing is done. Plain and simple.
Should you want more information stuffed into your return values, maybe you want this pattern instead:
Ah yes, the infamous if err != nil
pattern from Golang. This pattern annoys me for how frustrating this is, but it's common in Python. If you aren't familiar with this, here's a common pattern in Golang for comparison.
func doSomething() (int, error) {
return (300, nil)
}
func doErr() (int, error) {
return 0, errors.New("Fail")
}
func main() int {
val, err := doErr()
if err != nil {
return 100;
}
return 0;
}
The Golang logic is that exceptions are horrendously bad for programming language design, and are cumbersome at best and very annoying at worst. For experienced vets, this design choice is appreciated much more greatly than that of exception try/catch
. Golang is a compiler-based language and is able to infer errors similar to that of Java and Rust.
This multiple-value return is replicable in Python, thanks to the multi-assign feature of Python.
def do_func():
return 42, None
def do_fail():
return 0, None
wdef fail_with_msg():
return 0, "Your function failed"
def main():
result, err = fail_with_msg()
if err is not None:
print("Your program failed")
quit()
print(f"You got {result}")
The if err is not None
pattern can be reduce to a simpler form: if err:
.
def main():
result, err = fail_with_msg()
if err:
print("Your program failed")
quit()
print(f"You got {result}")
This works so long as the err
variable can prove itself to be some kind of meaningful data, meaning something that is rather... Not empty-ish looking. Strings with data, numbers, non-zero values, anything that can resolve to a True
value will pass, but anything that that is empty-ish, like None
, empty strings, the integer value zero, will resolve as False
. Check the bool(x)
function to see what returns True
or False
(because Python defines __bool__()
functions for most objects).
The value in multi-return is that it makes your functions easier to work with to obtain meaningful data about processes under the hood. Better to do this than throw random exceptions that can terminate a program entirely. It might be a little more man-power to add more return values, but this proves more useful than plainly passing None
as an error indicator. The more context you can pass around your libraries or APIs, the better, because things can break anywhere.
I am not a fan of exceptions, and I would love to avoid them altogether. If you're a Python person stuck with exception catching, maybe consider alternate strategies as to avoid overloading yourself with error unwrapping. The multi-value return paths are a nice way to alleviate yourself of front-facing exceptions that can occur, allowing you to design your back-end code to handle important, unavoidable exceptions. My ideal situation would look like this:
# back end function to handle exceptions
def do_job():
conn, err = networking.openListenPort(8088)
if err:
return 0, "Could not open up networking port"
try:
data, err = conn.listen()
if err is not None:
return 1, "Failed to listen on the port"
result, err = processing.startJob(data)
if err is not None:
return 2, "Failed to start job"
except IOError as ioerr:
return 3, f"I/O Error: {ioerr}"
except ConnectionError as conerr:
return 4, f"Connection error: {conerr}"
except Exception as err:
return 5, f"General error: {err}"
return 200, None
Then, on your front end:
from your_lib import do_job
def front_facing_code():
result, err = do_job()
if err is not None:
print(f"Error: {err}")
quit()
print(f"Result: {result}")
This hybrid approach is ideal, and I would rather leave the exceptions to somewhere behind the scenes, and make them less invasive and annoying surprises. If you have another approach, let me know, as this all comes down to preference, but the try/catch
meta of coding is certainly getting stale real fast.