Weird symbols in IPython Notebook - azure

Question

I use cyrillic symbols in my IPython notebooks. It works fine when I work in ML studio.
But when I download notebooks and open them (for example on http://try.jupyter.org ), I see strange characters.
Bad notebook (created on Azure ML Studio):
{"nbformat_minor": 0, "cells": [{"source": "\u00d1\u0082\u00d0\u00b5\u00d1\u0081\u00d1\u0082", "cell_type": "markdown", "metadata": {"collapsed": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.11", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}
$ file bad.ipynb
bad.ipynb: ASCII text, with very long lines, with no line terminators
"Good" version, created on http://try.jupyter.org:
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"тест"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
$ file good.ipynb
good.ipynb: UTF-8 Unicode text

Answer 1

I have done some lab, and found out that your json is encoded into utf-8. For your case, it's simple to get the real content back. See the code below:
Python 3.x
a = '{"nbformat_minor": 0, "cells": [{"source": "\u00d1\u0082\u00d0\u00b5\u00d1\u0081\u00d1\u0082", "cell_type": "markdown", "metadata": {"collapsed": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.11", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}'
result = a.encode('latin-1').decode("utf-8")
Python 2.x
a = '{"nbformat_minor": 0, "cells": [{"source": "\u00d1\u0082\u00d0\u00b5\u00d1\u0081\u00d1\u0082", "cell_type": "markdown", "metadata": {"collapsed": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.11", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}'
result = a.decode('unicode-escape').encode("latin-1")
This piece of code may not work for some other cases, because 'latin-1' does not cover all 0-255 characters. Hence, I am still looking for a better encoding for this kind of things.