The exact value that you set _XOPEN_SOURCE to doesn't matter. Mono is incorrectly defining the _XOPEN_SOURCE_EXTENDED macro:
case "${host}" in
*solaris* )
AC_MSG_CHECKING(for Solaris XPG4 support)
if test -f /usr/lib/libxnet.so; then
CPPFLAGS="$CPPFLAGS -D_XOPEN_SOURCE=600"
CPPFLAGS="$CPPFLAGS -D__EXTENSIONS__"
CPPFLAGS="$CPPFLAGS -D_XOPEN_SOURCE_EXTENDED=1" <-- WRONG
LIBS="$LIBS -lxnet"
The _XOPEN_SOURCE_EXTENDED macro does not exist in either the POSIX 6 or the POSIX 7 standard.
Even Linux agrees that _XOPEN_SOURCE_EXTENDED should not be defined here. Per the Linux feature_test_macros man page:
_XOPEN_SOURCE_EXTENDED
If this macro is defined, and _XOPEN_SOURCE is defined, then expose definitions corresponding to the XPG4v2 (SUSv1) UNIX extensions (UNIX 95). Defining _XOPEN_SOURCE with a value of 500 or more also produces the same effect as defining _XOPEN_SOURCE_EXTENDED. Use of _XOPEN_SOURCE_EXTENDED in new source code should be avoided.
Since defining _XOPEN_SOURCE with a value of 500 or more has the same effect as defining _XOPEN_SOURCE_EXTENDED, the latter (obsolete) feature test macro is generally not described in the SYNOPSIS in man pages.
Note the precise wording:
If this macro (_XOPEN_SOURCE_EXTENDED) is defined, and _XOPEN_SOURCE is defined, then expose definitions corresponding to the XPG4v2 (SUSv1) UNIX extensions (UNIX 95). ...
Defining _XOPEN_SOURCE to any value while also defining _XOPEN_SOURCE_EXTENDED results in XPG4v2, and that's NOT the XPG6 necessary to get recursive mutexes.
You're likely running into this check in the Solaris 11 /usr/include/sys/feature_tests.h:
/*
* It is invalid to compile an XPG3, XPG4, XPG4v2, or XPG5 application
* using c99. The same is true for POSIX.1-1990, POSIX.2-1992, POSIX.1b,
* and POSIX.1c applications. Likewise, it is invalid to compile an XPG6
* or a POSIX.1-2001 application with anything other than a c99 or later
* compiler. Therefore, we force an error in both cases.
*/
#if defined(_STDC_C99) && (defined(__XOPEN_OR_POSIX) && !defined(_XPG6))
#error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
and pre-2001 POSIX applications"
#elif !defined(_STDC_C99) && \
(defined(__XOPEN_OR_POSIX) && defined(_XPG6))
#error "Compiler or options invalid; UNIX 03 and POSIX.1-2001 applications \
require the use of c99"
#endif
_XPG6 gets defined earlier in the file, in this block:
/*
* Use of _XOPEN_SOURCE
*
* The following X/Open specifications are supported:
*
* X/Open Portability Guide, Issue 3 (XPG3)
* X/Open CAE Specification, Issue 4 (XPG4)
* X/Open CAE Specification, Issue 4, Version 2 (XPG4v2)
* X/Open CAE Specification, Issue 5 (XPG5)
* Open Group Technical Standard, Issue 6 (XPG6), also referred to as
* IEEE Std. 1003.1-2001 and ISO/IEC 9945:2002.
*
* XPG4v2 is also referred to as UNIX 95 (SUS or SUSv1).
* XPG5 is also referred to as UNIX 98 or the Single Unix Specification,
* Version 2 (SUSv2)
* XPG6 is the result of a merge of the X/Open and POSIX specifications
* and as such is also referred to as IEEE Std. 1003.1-2001 in
* addition to UNIX 03 and SUSv3.
*
* When writing a conforming X/Open application, as per the specification
* requirements, the appropriate feature test macros must be defined at
* compile time. These are as follows. For more info, see standards(5).
*
* Feature Test Macro Specification
* ------------------------------------------------ -------------
* _XOPEN_SOURCE XPG3
* _XOPEN_SOURCE && _XOPEN_VERSION = 4 XPG4
* _XOPEN_SOURCE && _XOPEN_SOURCE_EXTENDED = 1 XPG4v2
* _XOPEN_SOURCE = 500 XPG5
* _XOPEN_SOURCE = 600 (or POSIX_C_SOURCE=200112L) XPG6
*
* In order to simplify the guards within the headers, the following
* implementation private test macros have been created. Applications
* must NOT use these private test macros as unexpected results will
* occur.
*
* Note that in general, the use of these private macros is cumulative.
* For example, the use of _XPG3 with no other restrictions on the X/Open
* namespace will make the symbols visible for XPG3 through XPG6
* compilation environments. The use of _XPG4_2 with no other X/Open
* namespace restrictions indicates that the symbols were introduced in
* XPG4v2 and are therefore visible for XPG4v2 through XPG6 compilation
* environments, but not for XPG3 or XPG4 compilation environments.
*
* _XPG3 X/Open Portability Guide, Issue 3 (XPG3)
* _XPG4 X/Open CAE Specification, Issue 4 (XPG4)
* _XPG4_2 X/Open CAE Specification, Issue 4, Version 2 (XPG4v2/UNIX 95/SUS)
* _XPG5 X/Open CAE Specification, Issue 5 (XPG5/UNIX 98/SUSv2)
* _XPG6 Open Group Technical Standard, Issue 6 (XPG6/UNIX 03/SUSv3)
*/
/* X/Open Portability Guide, Issue 3 */
#if defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE - 0 < 500) && \
(_XOPEN_VERSION - 0 < 4) && !defined(_XOPEN_SOURCE_EXTENDED)
#define _XPG3
/* X/Open CAE Specification, Issue 4 */
#elif (defined(_XOPEN_SOURCE) && _XOPEN_VERSION - 0 == 4)
#define _XPG4
#define _XPG3
/* X/Open CAE Specification, Issue 4, Version 2 */
#elif (defined(_XOPEN_SOURCE) && _XOPEN_SOURCE_EXTENDED - 0 == 1)
#define _XPG4_2
#define _XPG4
#define _XPG3
/* X/Open CAE Specification, Issue 5 */
#elif (_XOPEN_SOURCE - 0 == 500)
#define _XPG5
#define _XPG4_2
#define _XPG4
#define _XPG3
#undef _POSIX_C_SOURCE
#define _POSIX_C_SOURCE 199506L
/* Open Group Technical Standard , Issue 6 */
#elif (_XOPEN_SOURCE - 0 == 600) || (_POSIX_C_SOURCE - 0 == 200112L)
#define _XPG6
#define _XPG5
#define _XPG4_2
#define _XPG4
#define _XPG3
#undef _POSIX_C_SOURCE
#define _POSIX_C_SOURCE 200112L
#undef _XOPEN_SOURCE
#define _XOPEN_SOURCE 600
#endif
