OpenBSD manual page server

Manual Page Search Parameters
RSA_GET_EX_NEW_INDEX(3) Library Functions Manual RSA_GET_EX_NEW_INDEX(3)

RSA_get_ex_new_index, RSA_set_ex_data, RSA_get_ex_data — add application specific data to RSA objects

#include <openssl/rsa.h>

int
RSA_get_ex_new_index(long argl, void *argp, CRYPTO_EX_new *new_func, CRYPTO_EX_dup *dup_func, CRYPTO_EX_free *free_func);

int
RSA_set_ex_data(RSA *rsa, int idx, void *data);

void *
RSA_get_ex_data(RSA *rsa, int idx);

The following parent objects can have application specific data called “ex_data” attached to them: BIO, DH, DSA, EC_KEY, RSA, SSL, SSL_CTX, SSL_SESSION, UI, X509, X509_STORE, and X509_STORE_CTX. The present manual page documents the related API functions taking the RSA object type as an example. The functions for the other object types work in exactly the same way: just replace the string "RSA" with the name of the respective object type throughout the rest of this manual page.

By default, each individual RSA object can store one void * pointing to application specific data. That specific pointer is identified by an idx argument of 0.

() reserves the next consecutive idx argument, enabling storage of one additional void * per RSA object. It is typically called at program startup. It can be called more than once if some RSA objects need to store more than two application specific pointers. Reserving an additional index for one parent object type, for example for RSA, does not change the numbers of indices that can be used with any other parent object type.

It is strongly recommended to always pass three NULL pointers for the arguments new_func, dup_func, and free_func. When following this recommendation, the arguments argl and argp are ignored; conventionally, passing 0 and NULL is recommended. Because using them is discouraged, the three function callback types are only documented in the low-level CRYPTO_EX_new(3) manual page.

() stores the data pointer as application specific data at the given idx in the given rsa object. The meaning of the data pointed to is up to the application. The caller retains ownership of the data and is responsible for freeing it when neither the caller nor the rsa object need it any longer. Any other pointer that was previously stored at the same idx in the same rsa object is silently overwritten. Passing a NULL pointer for the data argument is valid and indicates that no application specific data currently needs to be stored at the given idx.

() retrieves the last pointer that was stored using RSA_set_ex_data() at the given idx in the given rsa object.

RSA_get_ex_new_index() returns a new index equal to or greater than 1 or -1 if memory allocation fails.

RSA_set_ex_data() returns 1 on success or 0 if memory allocation fails.

RSA_get_ex_data() returns the application specific data or NULL if rsa does not contain application specific data at the given idx.

After failure of RSA_get_ex_new_index() or RSA_set_ex_data(), the following diagnostic can be retrieved with ERR_get_error(3), ERR_GET_REASON(3), and ERR_reason_error_string(3):

"malloc failure"
Memory allocation failed.

In a few unusual failure cases, ERR_get_error(3) may report different errors caused by OPENSSL_init_crypto(3) or even none at all.

RSA_get_ex_data() does not distinguish success from failure. Consequently, after RSA_get_ex_data() returns NULL, ERR_get_error(3) returns 0 unless there is still an earlier error in the queue.

BIO_set_ex_data(3), CRYPTO_set_ex_data(3), DH_set_ex_data(3), DSA_set_ex_data(3), RSA_new(3), SSL_CTX_set_ex_data(3), SSL_SESSION_set_ex_data(3), SSL_set_ex_data(3), X509_STORE_CTX_set_ex_data(3), X509_STORE_set_ex_data(3)

These functions first appeared in SSLeay 0.9.0 and have been available since OpenBSD 2.4.

A relatively small minority of application programs attempt to change the API contract such that RSA_set_ex_data() transfers ownership of the data to the rsa object. They do this by providing a free_func that calls free(3) or higher-level *_free() functions on the data and sometimes also attempt additional cleanup work as a side effect.

This practice is discouraged for several reasons:

  1. Due to a massive design mistake in the low-level API function CRYPTO_free_ex_data(3), this practice creates a possibility that RSA_free(3) may fail due to memory allocation failure, consequently leaking the memory containing the application specific data and silently skipping any additional cleanup work the free_func was supposed to do, leaving the application in an undetectably inconsistent state. Arguably, leaking additional memory while trying to free some is most unfortunate especially when the program is already starved for memory.
  2. This practice introduces a risk of use-after-free and double-free bugs in case the rsa object gets destructed while a caller of RSA_set_ex_data() or RSA_get_ex_data() still holds a data pointer. No such risk exists when no free_func is installed.
  3. Attempting additional cleanup work in free_func is an even worse idea because free_func is unable to report any issues it might detect while doing that work. Instead, if any additional cleanup work is needed, it is recommended that the calling code takes care of that before calling RSA_free(3).

Even fewer application programs install a new_func that allocates memory and stores a pointer to it in the rsa object by calling CRYPTO_set_ex_data(3). That is useless because new_func does not have access to any useful information it could store in such memory and because the default return value of NULL from RSA_get_ex_data() is sufficient to indicate that no application specific data has been stored yet. In addition, allocating memory in new_func is also inadvisable because it introduces an additional responsibility for callers of RSA_set_ex_data() to always call RSA_get_ex_data() first, even when it is the first time the application wants to set application specific data in a particular rsa object, and to either modify whatever RSA_get_ex_data() returns or to free it before calling RSA_set_ex_data(). If that is forgotten, a memory leak results.

Consequently, allocating any required memory is better left to the application code that calls RSA_set_ex_data().

Installing a dup_func is often seen in combination with installing a free_func, for obvious reasons. It is rarely useful because for most parent object types that support ex_data, including for RSA, the library does not provide a copying API function in the first place, and even where copying functions exist, they tend to be fragile and error-prone. When a new object is needed, it is usually advisable to construct it from scratch whenever possible, rather than attempting a copy operation.

On top of that, if dup_func fails, for example because of a memory allocation failure, the failure is neither reported nor detectable in any way, leaving the new parent object with incomplete data and potentially in an inconsistent state.

If RSA_set_ex_data() fails, recovery is very difficult. In particular, calling RSA_free(3) on the parent rsa object right afterwards is likely to also hit a memory allocation failure, leaking all memory internally allocated by all earlier calls of RSA_set_ex_data() on rsa rather than freeing that memory. In order to recover, the application program would have to free a sufficient amount of memory before calling RSA_free(3), which will rarely be feasible. Consequently, after a failure of RSA_set_ex_data(), terminating the program is likely the only reasonable option.

If RSA_set_ex_data() is called with an idx argument greater than the last one previously returned from RSA_get_ex_new_index(), it may still succeed, and though that is not guaranteed by the API, retrieving the data from such a bogus idx may even be possible with RSA_get_ex_data(), hiding the bug in the application program that caused passing the bogus idx to RSA_set_ex_data() in the first place.

If the bogus idx argument is large, RSA_set_ex_data() may uselessly allocate a large amount of memory. Calling RSA_free(3) on the parent rsa object is the only way to recover that memory.

If the bogus idx argument is very large, RSA_set_ex_data() is likely to cause a significant delay before eventually failing due to memory exhaustion. It is likely to return without releasing the memory already allocated, causing any subsequent attempt to allocate memory for other purposes to fail, too. In this situation, what was said above about failure of RSA_set_ex_data() applies, so terminating the program is likely the only reasonable option.

November 19, 2023 OpenBSD-current