Returned content not encoded automatically after changing Content-Type - cherrypy

Question

Given a resource with multiple representations (media types) that doesn't use custom CherryPy "tools" to handle the interpretation of the "Accept" HTTP header and the serialization of the response entity body, CherryPy raises the following ValueError exception upon returning the content from the page handler after setting the "Content-Type" HTTP header appropriately, but for certain media types only:
ValueError: Page handlers MUST return bytes. Use tools.encode if you wish to return unicode.
Example:
contentType = tools.accept.callable(media = ['application/json', 'text/html'])
if contentType == 'application/json':
return json.dumps(studies)
elif contentType == 'text/html':
...
This works for both media types, although the JSON representation will be wrongly declared as HTML (the default).
contentType = tools.accept.callable(media = ['application/json', 'text/html'])
response.headers['Content-Type'] = "{mediaType}; charset=utf-8".format(mediaType = contentType)
if contentType == 'application/json':
return json.dumps(studies)
elif contentType == 'text/html':
...
Here the exception above is raised when returning the JSON content as a string.
Attempts to ensure tools.encode is indeed enabled and setting tools.encode.encoding to utf-8 explicitly (even though it is the default) fail. Things work out for the HTML representation, so it seems like it should work for the JSON representation as well.
Currently the documentation for tools.encode seems rather sparse so which is the best approach to take is not immediately obvious.

Answer 1

The CherryPy encode tool automatically encodes content only when the top-level media type is text (text/*).
There is a way to control this with the encode.text_only setting, but it is global and so might introduce issues when returning content that really shouldn't be encoded. As of this writing, an open issue tracks a feature request for more granular control over this behavior: #1123.
For this reason, the most appropriate way to resolve this in this particular situation is to encode the content manually:
contentType = tools.accept.callable(media = ['application/json', 'text/html'])
response.headers['Content-Type'] = "{mediaType}; charset=utf-8".format(mediaType = contentType)
if contentType == 'application/json':
return json.dumps(studies).encode('utf-8')
elif contentType == 'text/html':
...