[base] Reject combinations of incompatible `FT_OPEN_XXX` flags.

The three modes are mutually exclusive, and the documentation of the
`FT_OPEN_XXX` constants notes this.  However, there was no check to
validate this in the code, and the documentation on `FT_Open_Args`
claimed that the corresponding bits were checked in a well-defined
order, implying it was valid (if useless) to specify more than one.
Ironically, this documented order did not agree with the actual
code, so it could not be relied upon; hopefully, nobody did this and
nobody will be hurt by the new validation.

Even if multiple mode bits were allowed, they could cause memory
leaks: if both `FT_OPEN_STREAM` and `stream` are set along with
either `FT_OPEN_MEMORY` or `FT_OPEN_PATHNAME`, then `FT_Stream_New`
allocated a new stream but `FT_Open_Face` marked it as an 'external'
stream, so the stream object was never released.

* src/base/ftobjs.c (FT_Stream_New): Reject incompatible
`FT_OPEN_XXX` flags.

ChangeLog

@@ -1,3 +1,25 @@
+2021-07-13  Oleg Oshmyan  <chortos@inbox.lv>
+
+	[base] Reject combinations of incompatible `FT_OPEN_XXX` flags.
+
+	The three modes are mutually exclusive, and the documentation of the
+	`FT_OPEN_XXX` constants notes this.  However, there was no check to
+	validate this in the code, and the documentation on `FT_Open_Args`
+	claimed that the corresponding bits were checked in a well-defined
+	order, implying it was valid (if useless) to specify more than one.
+	Ironically, this documented order did not agree with the actual
+	code, so it could not be relied upon; hopefully, nobody did this and
+	nobody will be hurt by the new validation.
+
+	Even if multiple mode bits were allowed, they could cause memory
+	leaks: if both `FT_OPEN_STREAM` and `stream` are set along with
+	either `FT_OPEN_MEMORY` or `FT_OPEN_PATHNAME`, then `FT_Stream_New`
+	allocated a new stream but `FT_Open_Face` marked it as an 'external'
+	stream, so the stream object was never released.
+
+	* src/base/ftobjs.c (FT_Stream_New): Reject incompatible
+	`FT_OPEN_XXX` flags.
+
 2021-07-12  Alex Richardson  <Alexander.Richardson@cl.cam.ac.uk>
 
 	* meson.build: Fix build for other UNIX systems (e.g., FreeBSD).

include/freetype/freetype.h

@@ -2113,8 +2113,7 @@ FT_BEGIN_HEADER
    *     Extra parameters passed to the font driver when opening a new face.
    *
    * @note:
-   *   The stream type is determined by the contents of `flags` that are
-   *   tested in the following order by @FT_Open_Face:
+   *   The stream type is determined by the contents of `flags`:
    *
    *   If the @FT_OPEN_MEMORY bit is set, assume that this is a memory file
    *   of `memory_size` bytes, located at `memory_address`.  The data are not
@@ -2127,6 +2126,9 @@ FT_BEGIN_HEADER
    *   Otherwise, if the @FT_OPEN_PATHNAME bit is set, assume that this is a
    *   normal file and use `pathname` to open it.
    *
+   *   If none of the above bits are set or if multiple are set at the same
+   *   time, the flags are invalid and @FT_Open_Face fails.
+   *
    *   If the @FT_OPEN_DRIVER bit is set, @FT_Open_Face only tries to open
    *   the file with the driver whose handler is in `driver`.
    *

src/base/ftobjs.c

@@ -197,6 +197,7 @@
     FT_Error   error;
     FT_Memory  memory;
     FT_Stream  stream = NULL;
+    FT_UInt    mode;
 
 
     *astream = NULL;
@@ -208,13 +209,15 @@
       return FT_THROW( Invalid_Argument );
 
     memory = library->memory;
+    mode   = args->flags &
+               ( FT_OPEN_MEMORY | FT_OPEN_STREAM | FT_OPEN_PATHNAME );
 
     if ( FT_NEW( stream ) )
       goto Exit;
 
     stream->memory = memory;
 
-    if ( args->flags & FT_OPEN_MEMORY )
+    if ( mode == FT_OPEN_MEMORY )
     {
       /* create a memory-based stream */
       FT_Stream_OpenMemory( stream,
@@ -224,13 +227,13 @@
 
 #ifndef FT_CONFIG_OPTION_DISABLE_STREAM_SUPPORT
 
-    else if ( args->flags & FT_OPEN_PATHNAME )
+    else if ( mode == FT_OPEN_PATHNAME )
     {
       /* create a normal system stream */
       error = FT_Stream_Open( stream, args->pathname );
       stream->pathname.pointer = args->pathname;
     }
-    else if ( ( args->flags & FT_OPEN_STREAM ) && args->stream )
+    else if ( ( mode == FT_OPEN_STREAM ) && args->stream )
     {
       /* use an existing, user-provided stream */